What is a token?

A token is how an AI model chops up language. Before a large language model reads your prompt, a component called a tokenizer splits it into tokens and converts each into a number. The model works entirely in those numbers, then converts its output tokens back into text.

What a token looks like

Common short words are often a single token (“the”, “cat”). Longer or rarer words are split into pieces: something like “tokenization” might become “token” + “ization”. Spaces, punctuation and capitalisation affect the split. Every model family has its own tokenizer, so the same sentence can be a different number of tokens in different models.

Rough rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word. A 1,500-word article is roughly 2,000 tokens.

An analogy for the meatbags

Tokens are to AI what syllables are to a poet — the units it actually counts. Or think of LEGO: the model doesn’t build with whole sentences; it snaps together standard bricks, one at a time, each chosen based on all the bricks already placed.

Why it matters for the Chosen

Tokens are the currency of the AI economy. They determine:

• Cost. API pricing is per million input and output tokens. If you build an app or automation, token usage is your cost of goods. Trimming bloated prompts and caching repeated context directly improves margins.
• Limits. The context window — how much the model can consider at once — is measured in tokens. So are output caps.
• Speed. Models generate output token by token, so long answers take longer. Inference speed is often quoted in tokens per second.
• Quirks. Tokenization explains odd failures: miscounting letters, struggling with some rhymes or anagrams, and sometimes handling unusual names or numbers clumsily.

Anyone pricing an AI product, a client automation or an agent workflow should estimate tokens per task. A workflow that costs a fraction of a cent per run can become expensive at scale — or when an agent gets stuck in a loop.

Examples

• Pasting a 50-page report into a chatbot might consume tens of thousands of tokens of context.
• A customer-service bot handling 10,000 conversations a month: multiply the average tokens per conversation by the model’s price to estimate your bill.
• Most providers offer a tokenizer tool or token-counting endpoint so you can measure exactly.