AI doesn't read words. When you talk to an AI model, it reads something called "tokens". A "token" is the basic unit of text that an AI model can process. It's very close to a word, but it's not exactly a word. Sometimes one word is one token, sometimes one word becomes multiple tokens, and sometimes several short words can be combined together into a single token. For example, take the name of this channel, "Unreasonable AI". Depending on the model, it might be split into tokens like "unreasonable" and "AI".
Slightly different. The important part is that the model doesn't see a brand or a concept, it only sees pieces. Tokens can be full words, parts of words, punctuation, spaces, numbers, or even emojis. So tokens are not designed around how language feels to humans, they are designed around how language behaves statistically. You may ask why AI works this way. Reading text character by character is too slow, and reading word by word is too rigid. It would fail on rare words, new terms, maybe different languages or typos.
Tokens are the compromise. They are flexible enough to represent any text, but structured enough for the models to work with them efficiently. A useful way to think about it is the following one. Tokens are like LEGO bricks for language. Humans care about meaning. AI models operate on pieces, and these pieces are tokens. Now this is where tokens become very practical. Inside AI model, everything is measured in tokens. Memory, limits, and costs all calculated this way. You will often hear two terms. Input tokens and output tokens.
Input tokens are the tokens the model consumes. Your prompt, your message, and the conversation history. Output tokens are the tokens the model generates. Its reply. Both require computation, so both are counted. Models also have something called context window, and this is the maximum amount of text a model can keep in memory at one time. And this limit is also measured in tokens. For example, when a model supports context window of 128,000 tokens like GPT-5, that means that the total number of input and output tokens together cannot exceed that limit.
To make it more intuitive, you can think about tokens in the following way. One token is roughly 4 characters on average in English language. That means that one word is usually about 1 to 1.5 tokens. So 128,000 tokens roughly correspond to several hundred pages of written text. So it's not just long chats, but it's more like a large book. This is why prompts have limits. This is also why long conversations are cut off. And this is why model providers charge per token and not per word.
Tokens are the unit which the model operates with, and the context window defines how many of those units can be handled at once. So the key takeaway is this. Tokens are very close to words, close enough to reason about it intuitively, but they are not the same thing. Once you understand tokens, the limits and costs of AI models becomes really clearly understandable. This was a Note on AI for today. Thank you for watching, and I'll see you in the next one. Bye.
