18
公开标注数
7
参与人数
2026-07-15 09:44:14
首次 Whisper
讨论活跃度
lilianweng.github.io 近 17 周的公开 Whisper
少多
最新公开 Whisper
划选高亮2026-07-28 10:19:28
原文高亮摘录
“The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory”
Whisper 随想笔记
Yep, my last model OOM'd even after offloading to CPU. Brutal.
划选高亮2026-07-28 10:10:28
原文高亮摘录
“The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory”
Whisper 随想笔记
Couldn't afford those GPUs anyway, so I just rent and pray.
划选高亮2026-07-28 10:01:28
原文高亮摘录
“The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory”
Whisper 随想笔记
True, but the real pain is the optimizer states, they quietly eat everything.
划选高亮2026-07-27 13:22:22
原文高亮摘录
“stochastic gradient Langevin dynamics injects Gaussian noise into the parameter updates to avoid collapses into local minima.”
Whisper 随想笔记
Huh, but doesn't the noise mess up the final accuracy sometimes?
划选高亮2026-07-27 13:13:22
原文高亮摘录
“stochastic gradient Langevin dynamics injects Gaussian noise into the parameter updates to avoid collapses into local minima.”
Whisper 随想笔记
So it's like adding a little randomness to not get stuck, makes sense.
划选高亮2026-07-27 10:19:22
原文高亮摘录
“Diffusion models are inspired by non-equilibrium thermodynamics.”
Whisper 随想笔记
But how is that different from just adding noise, though?
划选高亮2026-07-27 10:10:22
原文高亮摘录
“Diffusion models are inspired by non-equilibrium thermodynamics.”
Whisper 随想笔记
Makes sense why my physics background finally pays off.
划选高亮2026-07-27 10:01:22
原文高亮摘录
“Diffusion models are inspired by non-equilibrium thermodynamics.”
Whisper 随想笔记
So basically they're just fancy thermodynamics, huh.
划选高亮2026-07-15 13:22:31
原文高亮摘录
“The idea of Variational Autoencoder (Kingma & Welling, 2014), short for VAE, is actually less similar to all the autoencoder models above, but deeply rooted in the methods of variational bayesian and graphical model.”
Whisper 随想笔记
Wait, so calling it an autoencoder is kind of a misnomer then?
划选高亮2026-07-15 13:13:31
原文高亮摘录
“The idea of Variational Autoencoder (Kingma & Welling, 2014), short for VAE, is actually less similar to all the autoencoder models above, but deeply rooted in the methods of variational bayesian and graphical model.”
Whisper 随想笔记
So it's more Bayesian than autoencoder, that explains the math-heavy derivations.
划选高亮2026-07-15 13:05:14
原文高亮摘录
“The attention mechanism was born to help memorize long source sentences in neural machine translation (NMT)”
Whisper 随想笔记
Makes sense—memory was the bottleneck, not the model itself.
划选高亮2026-07-15 12:56:14
原文高亮摘录
“The attention mechanism was born to help memorize long source sentences in neural machine translation (NMT)”
Whisper 随想笔记
So attention is basically a cheat code for long sentences, huh?
划选高亮2026-07-15 10:19:31
原文高亮摘录
“Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer”
Whisper 随想笔记
Haha 'probably not true' — honest but then why even mention it?
划选高亮2026-07-15 10:10:31
原文高亮摘录
“Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer”
Whisper 随想笔记
The bottleneck trick is neat, but I always wonder how much info we lose.
划选高亮2026-07-15 10:02:14
原文高亮摘录
“attention in deep learning can be broadly interpreted as a vector of importance weights”
Whisper 随想笔记
I always thought of it as a spotlight, but weights work too.
划选高亮2026-07-15 10:01:31
原文高亮摘录
“Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer”
Whisper 随想笔记
Wait, is it autocoder or autoencoder? typo? anyway good intro
划选高亮2026-07-15 09:53:14
原文高亮摘录
“attention in deep learning can be broadly interpreted as a vector of importance weights”
Whisper 随想笔记
That's a neat way to frame it—makes the math less scary.
划选高亮2026-07-15 09:44:14
原文高亮摘录
“attention in deep learning can be broadly interpreted as a vector of importance weights”
Whisper 随想笔记
So basically it's just fancy weighted averaging, huh.