Token
A token is the smallest chunk of text a model works with — not exactly a word and not exactly a character. Common words are usually one token; rarer words get split into multiple tokens. This matters because model costs, speed, and limits are all measured in tokens, not words. It also explains some oddities: AI models sometimes struggle with exact character counts because they never see raw letters, only tokens.
Videos explaining this concept
E011Notes on AI
Tokens
AI models do not read words. They read tokens — the basic unit of text a model processes. A token is close to a word but not the same: one word can be one token, several tokens, or several words ca...
E012Notes on AI
Tokenization
Tokenization is the process of turning raw text into tokens before an AI model processes it. It is preprocessing, not thinking — the model only sees the resulting pieces.
E013Notes on AI
Why Typos Don't Matter
AI models don't read words — they read tokens, small chunks of characters. When you misspell a word, most of the token pieces remain the same. The pattern survives, and because meaning in language ...
E014Notes on AI
Context Window
The context window is the amount of information a model can see at one time. It's not memory — it's working space. The model can only reason about what is currently visible. At any moment, it recei...