Full Explanation
Tokenization is the process of turning raw text into tokens before an AI model processes it. It is preprocessing, not thinking — the model only sees the resulting pieces.
Tokenization is learned from training data, not designed by hand. It balances compression (efficient reuse of common patterns) and flexibility (representing rare words, new terms, different languages, and typos via subwords).
It is model-specific: different models use different tokenizers, so the same sentence can become different tokens and different counts. This explains why limits, costs, and odd behavior with numbers, emojis, or languages vary between providers.
Tokenization has nothing to do with meaning — the tokenizer only cuts text. A useful mental model: tokenization is the scissors; the model is the brain. Once you see this step, differences between models start making sense.
