Key-value caching

A decoder generates autoregressively: each new token is appended to the input and the model runs again. Here one masked self-attention layer is shown, simplified to QK^⊤V (no scaling or softmax). Because of the causal mask, row i of every matrix depends only on tokens 1..i, so everything in the green boxes is the same as in the previous step. The keys and values can be cached, and each step then only needs the newest token's query. Step through, then switch on the cache.