The Transformer Architecture, From Attention Up

Every large language model I've worked with — the one behind this site's chatbot, the ones I use daily — is a stack of the same block repeated dozens of times. If you understand that one block, you understand the architecture. Everything else is scale.

I'm going to build it up in the order it actually made sense to me: attention first, then multi-head, then the plumbing that makes a deep stack trainable.

The problem attention solves

Before transformers, sequence models read tokens one at a time and carried a hidden state forward. That's inherently sequential — token 500 has to wait for token 499 — and long-range dependencies get diluted along the way.

Attention throws that out. Instead of passing state down a chain, every token looks at every other token in parallel and pulls in whatever is relevant. "The animal didn't cross the street because it was tired" — attention lets it look back and weight animal heavily and street lightly.

Query, key, value

Each token's embedding is projected into three vectors: a query, a key, and a value. The mental model that stuck for me is a soft dictionary lookup:

• The query is what this token is looking for.
• Every token also exposes a key — what it offers.
• The match between a query and each key becomes a weight, and the output is the weighted sum of everyone's value.

The whole operation is one formula:

Attention(Q, K, V) = softmax(Q · Kᵀ / √dₖ) · V

Q · Kᵀ scores every query against every key. Dividing by the square root of the key dimension (√dₖ) keeps those scores from blowing up as dimensions grow — without it, softmax saturates and gradients die. Softmax turns the scores into weights that sum to one, and multiplying by V mixes the values accordingly.

Multi-head: several conversations at once

One attention operation learns one notion of "relevant." Language has many at once — syntax, coreference, tense. So the transformer runs several attention operations in parallel on lower-dimensional projections, then concatenates the results.

def multi_head_attention(x, heads):
    outs = []
    for head in heads:
        q, k, v = head.project(x)          # each head: its own Q/K/V
        outs.append(attention(q, k, v))
    return concat(outs) @ W_o              # merge heads back to model width

Each head is free to specialize. In practice you find heads that track the previous token, heads that follow punctuation, heads that resolve pronouns. Nobody programs that — it falls out of training.

Position: attention is order-blind

Here's the catch: the attention math is a set operation. Shuffle the tokens and the output shuffles with them, but the relationships are identical. "Dog bites man" and "man bites dog" would look the same. So we inject order explicitly with positional encodings added to the embeddings — either fixed sinusoids or learned vectors. Modern models often use rotary encodings that rotate Q and K by position, which generalizes better to longer sequences.

The block that gets stacked

A transformer layer is attention plus a small feed-forward network, each wrapped the same way:

x = x + Attention(LayerNorm(x))
x = x + FeedForward(LayerNorm(x))

Two details do the heavy lifting:

• Residual connections (x + ...): the signal can skip the sublayer entirely, so gradients flow cleanly through a 40-layer stack instead of vanishing.
• Layer normalization: keeps activations in a stable range regardless of depth.

The feed-forward network is just two linear layers with a nonlinearity — it expands to roughly 4× the model width, applies the activation, and projects back. It's where a lot of the model's raw capacity lives.

Encoder, decoder, or both

The original paper had an encoder (reads the whole input at once) and a decoder (generates left to right, and can't peek at future tokens thanks to a causal mask). The GPT-style models that dominate today are decoder-only: one masked stack that predicts the next token, over and over.

Why this shape won

It parallelizes across the sequence, so you can throw hardware at it. It has a short path between any two tokens, so long-range dependencies survive. And it scales predictably — more layers, more width, more data, better model. That last property is why "just make it bigger" has been a working strategy for years.

Once the block clicked, the frameworks stopped feeling like magic. When I debug a RAG pipeline or reason about context windows, I'm thinking in terms of this same attention-over-tokens picture — just at scale.