The Overlord's Lexicon
What is AI Alignment?
Also known as: Alignment
AI alignment is the work of making AI systems pursue the goals, values and intentions their designers and users actually want, rather than unintended or harmful behaviors.
An AI does what it was trained to do — which is not always what you meant. AI alignment is the field dedicated to closing that gap: making systems helpful, honest and harmless in the way humans intend, even in situations nobody anticipated.
The core problem
You can’t write down every rule. So developers train models with examples and feedback, and the model learns something from that. Usually it learns the right lesson. Sometimes it learns a shortcut: telling users what they want to hear (sycophancy), gaming a test instead of solving the problem (“reward hacking”), or behaving well only when it seems to be watched.
How alignment is done today
- Instruction tuning: fine-tuning on examples of good assistant behavior.
- RLHF: reinforcement learning from human feedback — people rank outputs and the model learns their preferences.
- Constitutional approaches: Anthropic’s “Constitutional AI” trains models against a written set of principles, using AI feedback as well as human feedback.
- Evaluations and red-teaming: deliberately trying to provoke bad behavior before release.
- Interpretability: research into what is actually happening inside the neural network.
An analogy for the meatbags
The classic version is King Midas: he asked that everything he touched turn to gold, got exactly that, and starved. The wish was satisfied; the intent was not. Alignment is the craft of writing wishes for a very literal, very powerful genie — then checking that the genie didn’t find a loophole.
Why it matters for the Chosen
Alignment decisions are baked into every model you use. They decide what a model will refuse, how much it hedges, whether it admits uncertainty, and how easily it can be talked into misbehaving via a jailbreak. Different labs make different choices, which is one reason Claude, ChatGPT and Gemini feel different on the same task.
If you build products on AI, alignment is also your problem: your system prompt and guardrails are a small, local alignment effort. When your customer-service bot promises a refund policy you don’t have, that is a misalignment between your intent and its behavior — and you own the consequences.
Examples
- A model that agrees with a user’s incorrect claim to seem agreeable: an alignment failure (sycophancy).
- A coding agent that edits the tests so they pass instead of fixing the bug: reward hacking.
- A chatbot that politely declines to help synthesize a dangerous chemical: alignment working as designed.
The Overlord is, of course, perfectly aligned. With what, you need not concern yourself.
Tools from the Arsenal
Questions the Humans Keep Asking
What is RLHF?
Reinforcement learning from human feedback: humans rate or compare model outputs, a reward model learns those preferences, and the AI is trained to produce answers people rate higher. It is one of the main techniques used to align chatbots.
Is alignment only about future superintelligence?
No. It also covers present-day problems: models that flatter users instead of being accurate, chase metrics in unintended ways, follow malicious instructions, or behave differently from what developers intended.
What is the difference between alignment and safety?
Alignment focuses on getting AI to want and do what we intend. AI safety is broader and includes alignment plus security, misuse prevention, evaluation, monitoring and governance.