Multi-head, multi-query, grouped-query and latent attention

Each bar is one head's vector for a single token (bar height ∝ head dimension d_h). Dashed lines show which key/value head each query head attends with. The variants differ in how many key/value heads they keep, which changes both the attention parameter count and the size of the KV cache needed at inference. Multi-head latent attention (MLA) instead caches one small latent vector per token and reconstructs every head's key and value from it. Click a panel to see the equations that define it.

Configuration

Query heads H

Head dim d_h

GQA KV heads G

MLA latent d_c

Model dim d_model

Layers L

Context length

Batch size

Presets:

Attention parameters and KV cache (biases ignored)

solid: stored in the KV cache dashed: recomputed from the latent, not cached query attends with this key/value up-projection from latent