The Whole Room Is Listening — Part 3
Transformer Encoders: Or, How Multi-Head Attention Listens to Everything Simultaneously Without Missing the Point
[This is Part 3. Part 1 covers autoencoders. Part 2 covers self-attention.]
At the end of Part 2, we had a problem: self-attention can look at every word from every other word, but it does not know where each word sits. It is like a very attentive listener who cannot remember the order in which things were said.
We also had a second problem, one Mathityahu raised on the way out.
“One head of attention,” he said, “is like one person listening. They might catch the subject-verb relationship. But they might miss the pronoun reference three sentences back. What you want is the whole room listening — each person picking up something different.”
This is multi-head attention. And combined with positional encoding, it is the foundation of the transformer encoder.
Problem 1: Position
A transformer has no inherent sense of order. The attention computation is a set of dot products — it treats position 1 and position 10 identically. Shuffle the tokens and the output is unchanged. For language this is fatal. “The dog bit the man” and “The man bit the dog” contain the same four words. Position is the only thing separating news from miracle.
The fix: before the first attention layer, add a small position-specific nudge to each word’s embedding. Word at position 3 gets a different nudge than the word at position 7. Now when Q and K are computed, position is already baked in.
“Add what, exactly?” asked Devorah.
A vector — the same length as the embedding — that is unique to each position. The model adds it component-by-component to the token embedding. After that, the same word appearing at two different positions produces two different vectors going into attention.
The hard part is designing those vectors. Two things are required:
- Every position must get a unique fingerprint — no two positions can be confused
- The fingerprint must generalize — a model trained on sequences of length 512 should handle length 1000 without seeing garbage
The simplest idea — just use the position number directly — fails both. Large positions produce large numbers that swamp the embedding values, and a model that saw positions 1–512 in training has no idea what to do with position 513.
The binary analogy
Consider how binary numbers count. The rightmost bit flips every step, the next bit every two steps, the next every four, and so on:
| Position | Bit 2 | Bit 1 | Bit 0 |
|---|---|---|---|
| 0 | 0 | 0 | 0 |
| 1 | 0 | 0 | 1 |
| 2 | 0 | 1 | 0 |
| 3 | 0 | 1 | 1 |
| 4 | 1 | 0 | 0 |
Every position gets a unique combination. And no bit ever blows up — they stay 0 or 1 regardless of how far you count.
Sinusoidal positional encoding is the continuous version of this idea. Instead of discrete bits flipping at different rates, use smooth sine and cosine waves cycling at different frequencies. Each pair of dimensions gets one frequency:
pos is the position in the sequence, i is the dimension pair index, d is the embedding size. The denominator grows with i, so:
- Low dimensions (small i): fast-cycling wave — flips sign every few positions, like the rightmost bit
- High dimensions (large i): slow-cycling wave — barely moves across an entire sentence, like the leftmost bit
The assignment of fast to low dimensions and slow to high is a convention — you could reverse it and the encoding would work just as well. What is not arbitrary is having both speeds. Fast waves are sensitive to small position differences: positions 7 and 8 look very different. But they loop — positions 7 and 7+period look identical. Slow waves never loop within a realistic sequence, so they separate distant positions that fast waves confuse. Neither alone is enough. Together they give you a fingerprint that is locally precise and globally unique.
Together, the 512 dimensions produce a combination of values that is unique at every position and bounded between −1 and 1 everywhere — so it never swamps the embeddings, and a frequency the model saw during training is still the same frequency at position 1000. Both constraints satisfied.
A concrete example
Take the sentence “The bubbe made matzah ball soup again.” — seven tokens, zero-indexed. “matzah” sits at position 3.
To keep the numbers readable, suppose the embedding has only 8 dimensions instead of 512. The positional encoding for position 3 is computed as:
| Dimension | Formula | Value |
|---|---|---|
| 0 | sin(3 / 10000⁰) | sin(3.000) = 0.14 |
| 1 | cos(3 / 10000⁰) | cos(3.000) = −0.99 |
| 2 | sin(3 / 10000^0.25) | sin(0.300) = 0.30 |
| 3 | cos(3 / 10000^0.25) | cos(0.300) = 0.95 |
| 4 | sin(3 / 10000^0.5) | sin(0.030) = 0.03 |
| 5 | cos(3 / 10000^0.5) | cos(0.030) = 1.00 |
| 6 | sin(3 / 10000^0.75) | sin(0.003) = 0.003 |
| 7 | cos(3 / 10000^0.75) | cos(0.003) = 1.00 |
Notice the pattern. Dimensions 0 and 1 are doing heavy oscillating — position 3 sits at a noticeable angle in a fast cycle. By dimension 4, the denominator is 100, so position 3 barely registers. By dimension 6, it is essentially zero. The fast dimensions distinguish nearby tokens; the slow ones help with tokens far apart.
Now suppose “matzah” has an embedding vector (its learned meaning):
embedding("matzah") = [ 0.80, −0.20, 0.50, 0.10, −0.30, 0.70, 0.20, −0.40 ]
Adding the positional encoding for position 3:
PE(position=3) = [ 0.14, −0.99, 0.30, 0.95, 0.03, 1.00, 0.003, 1.00 ]
+
input to attention = [ 0.94, −1.19, 0.80, 1.05, −0.27, 1.70, 0.203, 0.60 ]
If the same word “matzah” appeared at position 6 instead, the positional encoding would be different — different sin/cos values, different resulting vector. Same word, different position, different input to the attention layers. That is the whole point.
import torch
import numpy as np
def positional_encoding(max_len, d_model):
pe = torch.zeros(max_len, d_model)
position = torch.arange(0, max_len).unsqueeze(1).float()
div_term = torch.exp(
torch.arange(0, d_model, 2).float() * -(np.log(10000.0) / d_model)
)
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
return pe # shape: (max_len, d_model)
# Use it: add to embeddings before the attention layers
x = token_embeddings + positional_encoding(seq_len, d_model)
Here is what the encoding looks like across 60 positions and 64 dimensions — each row is one position’s fingerprint:
The fast oscillations on the left (low dimensions) and the slow gradients on the right (high dimensions) are visible. No two rows are the same. The model adds this matrix to the token embeddings before anything else — now every token carries both what it is and where it sits, baked into the same vector.
Problem 2: One Head Is Not Enough
A single attention head learns one type of relationship. In a sentence, there are many: subject-verb agreement, pronoun reference, modifier attachment, semantic similarity. One head cannot focus on all of them simultaneously.
Multi-head attention runs h independent attention heads in parallel, each with its own Q, K, V projections — each free to learn different relationships. Then it concatenates the outputs and projects back to the original dimension.
import torch.nn as nn
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
assert d_model % n_heads == 0
self.d_head = d_model // n_heads
self.n_heads = n_heads
self.W_qkv = nn.Linear(d_model, 3 * d_model, bias=False)
self.W_out = nn.Linear(d_model, d_model, bias=False)
def forward(self, x):
B, T, C = x.shape
qkv = self.W_qkv(x) # (B, T, 3*d_model)
q, k, v = qkv.split(C, dim=2)
# Split into heads: (B, n_heads, T, d_head)
def split_heads(t):
return t.view(B, T, self.n_heads, self.d_head).transpose(1, 2)
q, k, v = split_heads(q), split_heads(k), split_heads(v)
# Scaled dot-product attention per head
scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5)
weights = scores.softmax(dim=-1)
out = weights @ v # (B, n_heads, T, d_head)
# Concatenate heads and project
out = out.transpose(1, 2).contiguous().view(B, T, C)
return self.W_out(out)
Eight heads in a 512-dimensional model: each head gets 64 dimensions. Eight different questions, asked simultaneously, about every pair of tokens. One head might track syntax. Another semantics. Another long-range dependencies. None of them know what the others found.
The Full Encoder Block
One transformer encoder block combines everything: multi-head attention, a position-wise feedforward network, residual connections, and layer normalization.
The residual connections (shown in amber) add the block’s input to its output before normalization. Instead of learning a full transformation output = f(x), each block only learns the difference from its input: output = x + f(x). It only has to figure out what to change, not reconstruct everything from scratch.
“Why does that matter?” asked Devorah.
Two reasons. First: gradients. During training, the gradient flows backward through every layer. In a network with 96 layers, by the time the gradient reaches layer 1 it has been multiplied by a small number 96 times — it is essentially zero, and that layer learns nothing. The residual connection adds a direct highway: the gradient can skip straight through the addition without passing through f at all, so early layers still receive a meaningful signal.
Second: graceful degradation. If a block learns nothing useful — f(x) ≈ 0 — the output is still x. The signal passes through unchanged. Adding more layers can never make things worse. The network can be made as deep as needed and will only use the capacity it can actually train.
In PyTorch, using the built-in nn.MultiheadAttention:
import torch.nn as nn
class TransformerEncoderBlock(nn.Module):
def __init__(self, d_model, n_heads, d_ff, dropout=0.1):
super().__init__()
self.attention = nn.MultiheadAttention(
d_model, n_heads, dropout=dropout, batch_first=True
)
self.ff = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Linear(d_ff, d_model),
)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
self.dropout = nn.Dropout(dropout)
def forward(self, x):
# Multi-head self-attention + residual
attn_out, _ = self.attention(x, x, x)
x = self.norm1(x + self.dropout(attn_out))
# Feed-forward + residual
x = self.norm2(x + self.dropout(self.ff(x)))
return x
Stack N of these blocks, and you have a transformer encoder. BERT uses 12. GPT-2 uses 12. The basic block is the same; the scale differs.
Putting It Together: A Full Encoder
import torch
import numpy as np
class TransformerEncoder(nn.Module):
def __init__(self, vocab_size, d_model=128, n_heads=4,
n_layers=2, d_ff=256, max_len=64, dropout=0.1):
super().__init__()
self.embedding = nn.Embedding(vocab_size, d_model)
self.register_buffer('pe', self._make_pe(max_len, d_model))
self.layers = nn.ModuleList([
TransformerEncoderBlock(d_model, n_heads, d_ff, dropout)
for _ in range(n_layers)
])
self.norm = nn.LayerNorm(d_model)
def _make_pe(self, max_len, d_model):
pe = torch.zeros(max_len, d_model)
pos = torch.arange(max_len).unsqueeze(1).float()
div = torch.exp(torch.arange(0, d_model, 2).float()
* -(np.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(pos * div)
pe[:, 1::2] = torch.cos(pos * div)
return pe.unsqueeze(0) # (1, max_len, d_model)
def forward(self, token_ids):
x = self.embedding(token_ids) + self.pe[:, :token_ids.size(1)]
for layer in self.layers:
x = layer(x)
return self.norm(x)
# Quick test — no training, just shape verification
encoder = TransformerEncoder(vocab_size=1000, d_model=128, n_heads=4, n_layers=2)
tokens = torch.randint(0, 1000, (2, 20)) # batch=2, seq_len=20
output = encoder(tokens)
print(f"Input: {tokens.shape}") # (2, 20)
print(f"Output: {output.shape}") # (2, 20, 128) — one 128-dim vector per token
Each token in the output is a 128-dimensional vector that has been informed by every other token in the sequence. This is the encoded representation. A classifier head sitting on top of this can do sentiment analysis. A cross-attention layer connected to a decoder can do translation.
What BERT Actually Did
BERT (Bidirectional Encoder Representations from Transformers) took this architecture and trained it on a simple task: predict the missing words.
Take a sentence. Randomly mask 15% of the tokens. Train the encoder to predict what was masked, looking at the full context — left and right. Do this on the entire internet, for a very long time.
The resulting model develops representations that understand language deeply enough that with only a small amount of task-specific fine-tuning, it beats the previous state of the art on nearly every benchmark.
The encoder is not magic. It is attention plus position plus residuals plus scale. But scale, it turns out, is a large fraction of the magic.
The Through-Line
We started with Devorah’s shoeboxes.
An autoencoder taught us that you can compress information into a small space — and that the geometry of that compressed space is meaningful. But it treats inputs as bags of numbers, no order.
Self-attention taught us to look at everything at once — to let every token inform every other token, and to learn what to look at rather than being told. But a single head sees only one type of relationship, and position is invisible.
The transformer encoder brings it together: positional encoding gives every token a coordinate in the sequence; multi-head attention lets the model ask many questions simultaneously; residual connections and layer norm keep training stable; stacking the blocks builds depth.
“So it listens,” said Devorah, after Mathityahu had finished.
“It listens,” he confirmed. “To everything. At once.”
“Like Aunt Rivka.”
“Exactly like Aunt Rivka. But with better memory, and more weights.”
All code in this series runs on CPU. Full code: Jewpyter notebook repository.