Tokenization
Before a model can process text, it has to be split into discrete units — tokens. This lesson is a placeholder for how modern tokenizers work.
What's coming
- Word-level vs. subword vs. byte-level tokenization.
- Byte-pair encoding, and why it's the default for most LLMs.
- How tokenization choices affect context length and cost.