Lexical analysis - Wikipedia
5
公开标注数
5
参与人数
2026-07-16 10:04:30
首次 Whisper
本页的公开 Whisper
划选高亮2026-07-16 13:25:30
原文高亮摘录
“Regular expressions and the finite-state machines they generate are not powerful enough to handle recursive patterns”
Whisper 随想笔记
I mean, sure, but for 99% of real tasks regex gets the job done.
划选高亮2026-07-16 13:16:30
原文高亮摘录
“Regular expressions and the finite-state machines they generate are not powerful enough to handle recursive patterns”
Whisper 随想笔记
Yeah, that's why regex can't parse HTML, folks keep learning that the hard way.
划选高亮2026-07-16 10:22:30
原文高亮摘录
“A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models”
Whisper 随想笔记
Reminds me of my compilers class—tokens had fixed types, not embeddings.
划选高亮2026-07-16 10:13:30
原文高亮摘录
“A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models”
Whisper 随想笔记
But probabilistic tokens still work way better for text generation, so...
划选高亮2026-07-16 10:04:30
原文高亮摘录
“A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models”
Whisper 随想笔记
So LLM tokens are just random guesses compared to real lexing, huh?
分享本页 Whisper
短链接
https://domwhisper.com/s/2fc692aab634嵌入代码
<iframe src="https://domwhisper.com/embed/2fc692aab634" width="100%" height="480" style="border:0;border-radius:16px" loading="lazy"></iframe>