Lexical analysis - Wikipedia

Public whispers

Contributors

2026-07-16 10:04:30

First whispered

Public whispers on this page

Text Highlight2026-07-16 13:25:30

Original Highlight Excerpt

"Regular expressions and the finite-state machines they generate are not powerful enough to handle recursive patterns"

Whisper Note

I mean, sure, but for 99% of real tasks regex gets the job done.

Text Highlight2026-07-16 13:16:30

Original Highlight Excerpt

"Regular expressions and the finite-state machines they generate are not powerful enough to handle recursive patterns"

Whisper Note

Yeah, that's why regex can't parse HTML, folks keep learning that the hard way.

Text Highlight2026-07-16 10:22:30

Original Highlight Excerpt

"A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models"

Whisper Note

Reminds me of my compilers class—tokens had fixed types, not embeddings.

Text Highlight2026-07-16 10:13:30

Original Highlight Excerpt

"A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models"

Whisper Note

But probabilistic tokens still work way better for text generation, so...

Text Highlight2026-07-16 10:04:30

Original Highlight Excerpt

"A lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models"

Whisper Note

So LLM tokens are just random guesses compared to real lexing, huh?

Short link

https://domwhisper.com/s/2fc692aab634

Embed snippet

<iframe src="https://domwhisper.com/embed/2fc692aab634" width="100%" height="480" style="border:0;border-radius:16px" loading="lazy"></iframe>

Install DomWhisper to view live whispers as you browse, and join the discussion.