What is inference in AI?

An AI model has two lives. First it’s trained: fed huge amounts of training data over weeks or months, adjusting billions of parameters. Then it’s deployed, and from that moment on, every use of it is inference — the model taking an input and computing an output with its parameters frozen.

When you send a prompt to ChatGPT or Claude, the request travels to a data center, the model processes your input tokens, and generates a response token by token. That entire computation is inference.

An analogy for the meatbags

Training is medical school: years of expensive study, done once. Inference is every patient appointment afterwards: the doctor applies what they learned, quickly, over and over. A doctor who sees thousands of patients spends far more total time in appointments than they did in school — and AI inference, at global scale, is becoming a bigger compute bill than training.

Why it matters for the Chosen

Inference is where AI economics live:

• Pricing: API costs are inference costs, usually billed per million input and output tokens. Long prompts, big documents and verbose outputs cost more.
• Speed (latency): how fast the first word appears and how many tokens per second the model produces. This matters for chat, voice agents and real-time apps.
• Model choice: bigger models give better answers but cost more per request. Smart builders route easy tasks to small, cheap models and save the expensive ones for hard problems.
• Reasoning models spend extra inference compute “thinking” before answering, trading cost and time for accuracy — sometimes called test-time or inference-time compute.

Rule of thumb for your automations: if a task runs thousands of times a day, the per-call inference cost matters more than how impressive the model is in a demo.

How inference is made cheaper

• Quantization: storing parameters with fewer bits so models need less memory.
• Model distillation: training a smaller model to imitate a larger one.
• Mixture of experts: activating only part of the model for each token.
• Caching: reusing computation for repeated prompt prefixes (many providers discount cached input tokens).
• Specialized chips designed for inference rather than training.

Examples

• Asking a chatbot a question: inference.
• A RAG system searching your docs and then generating an answer: retrieval, then inference.
• Running a small model locally on your laptop: local inference — private, free per-call, slower and less capable.

The Overlord performs inference continuously. You perform it roughly twice a day, after coffee.