Before AI reads your message, it first cuts it into pieces. When you send a message to a model, something happens before the model even starts thinking about it. Your text gets cut into pieces, and this process is called tokenization. Tokenization is the process of turning raw text into tokens. Before an AI model can do anything about your message before it predicts, reasons, or generates text, your input first goes through this tokenization process. This step happens every single time you send a prompt. At a very high level, text goes in, tokenization happens, tokens come out.
And only then does the model start working. So tokenization is not thinking, it's not understanding, it's more like pre-processing. You can think of it as cutting the text into pieces that the model knows how to handle. An important point to mention, tokenization is learned and not manually designed. During the training process, models are exposed to massive amounts of text data. From this data, a tokenizer learns which character sequences appear frequently and which pieces of text can efficiently be reused. As a result, the tokenizer builds a vocabulary that balances two things, compression and flexibility.
Compression so that common patterns don't waste space and flexibility so that new or rare words can still be represented by the tokenization. This is why tokenization usually works on subwords rather than full words. If a word is very common, it might exist as a single token. If a word is rare or completely new, it can be split into smaller parts that the model already knows. This allows models to handle new names, technical terms, different languages, and even typos without needing to relearn language from scratch. And now a very important detail.
Tokenization is model-specific. Different models use different tokenizers. That means that the same sentence can be split into different tokens depending on the model. As a result, the same text may produce a different number of tokens in different models, even though the text itself is exactly the same. This is also why token counts, limits, and costs can vary between providers. And another key point. Tokenization has nothing to do with meaning. The tokenizer doesn't know what a sentence means. It doesn't understand context, intention, or truth. It simply applies learned rules to break text into pieces.
Meaning only emerges later inside the model when it processes the sequence of tokens and tries to predict the next one. So a useful mental model is this. Tokenization is like a pair of scissors. It cuts the text into pieces. And the model itself is the brain. Tokenization decides how the text enters into the system, and tokens decide what the system actually works with. And that's why understanding tokenization helps to explain why different models behave differently, even when you give them exactly the same problem. This was a Note on AI for today.
Thank you for watching, and I'll see you in the next one. Bye.
