PreviewYou're watching a preview: the opening of this lecture. The full lecture and the rest of the course need full access.Get full access
Foundations
Tokens
PreviewLesson 1 of 21 · 25 min
LLMs never see raw text, only tokens. This lecture shows how a tokenizer splits text with Byte Pair Encoding, and why that drives cost.
- Why words split in strange places
- Why Hindi and other languages use more tokens
- Why ChatGPT, Claude and DeepSeek tokenize differently
The full lecture, notes, demo code and quiz
Included with full access to the course.