Knowledge distillation - Wikipedia
6
Public whispers
6
Contributors
2026-07-18 10:02:04
First whispered
Public whispers on this page
Text Highlight2026-07-18 16:26:04
Original Highlight Excerpt
"model compression is a special case of knowledge distillation"
Whisper Note
So compression is just distillation with extra steps, got it.
Text Highlight2026-07-18 13:23:04
Original Highlight Excerpt
"high temperature increases the entropy of the output"
Whisper Note
But doesn't that just blur the signal? I'd rather keep crisp targets.
Text Highlight2026-07-18 13:14:04
Original Highlight Excerpt
"high temperature increases the entropy of the output"
Whisper Note
So higher temp basically makes the model less sure but more teachable, got it.
Text Highlight2026-07-18 10:20:04
Original Highlight Excerpt
"process of transferring knowledge from a large model to a smaller one"
Whisper Note
We use this at work to shrink our models for edge devices, works great.
Text Highlight2026-07-18 10:11:04
Original Highlight Excerpt
"process of transferring knowledge from a large model to a smaller one"
Whisper Note
Doesn't the smaller one lose some nuance though? Seems inevitable.
Text Highlight2026-07-18 10:02:04
Original Highlight Excerpt
"process of transferring knowledge from a large model to a smaller one"
Whisper Note
So it's like a big brain teaching a small brain its tricks, neat.
Share this page's whispers
Short link
https://domwhisper.com/s/618b8b2dee31Embed snippet
<iframe src="https://domwhisper.com/embed/618b8b2dee31" width="100%" height="480" style="border:0;border-radius:16px" loading="lazy"></iframe>See what people are discussing on en.wikipedia.org
Install DomWhisper to view live whispers as you browse, and join the discussion.
Get the extension