What is a transformer?

The transformer is the architecture — the blueprint for how the neural network is wired — behind ChatGPT, Claude, Gemini, Llama and nearly every other large language model. It’s the “T” in GPT. If modern AI is a building, the transformer is the steel frame.

How it works

Before transformers, language models mostly read text one word at a time, in order, which made them slow to train and forgetful over long passages. The transformer, introduced in the 2017 Google paper Attention Is All You Need, changed two things:

1. Attention. For every token, the model computes how strongly it should “attend” to every other token. In “The trophy didn’t fit in the suitcase because it was too big,” attention helps the model link “it” to “trophy.” Many attention “heads” run in parallel, each picking up different relationships — grammar, references, topic.
2. Parallelism. Because attention looks at the whole sequence at once rather than step by step, training can be spread across thousands of chips. That made it practical to train enormous models on enormous datasets, which is what unlocked today’s capabilities.

Stack dozens of these attention layers, train on a vast corpus, and you get a model that captures remarkably subtle patterns in language.

An analogy for the meatbags

Older models read a sentence like someone listening to a long voicemail, struggling to remember the start by the end. A transformer reads like someone with the whole page in front of them, highlighter in hand, drawing lines between every word and the other words that explain it.

Why it matters for the Chosen

You don’t need to implement attention to make money with AI. But knowing what a transformer is helps you:

• Cut through hype. Most “revolutionary new AI” products are transformer models from a few major labs, wrapped in an interface — see AI wrapper.
• Understand limits. Attention cost grows quickly with input length, which is part of why context windows have limits and long inputs cost more.
• Follow the field. Variants like mixture of experts and techniques like quantization are all about making transformers cheaper and faster.

Examples

Text: GPT, Claude, Gemini, Llama. Images: vision transformers used in computer vision. Generation: many diffusion image and video models now use transformer backbones. Science: transformer-based models have been applied to predicting protein structures.