The Mechanics of Flash-Decoding
When you give an AI a massive prompt, like an entire book or thousands of lines of code, things get messy. The system hits a speed bump. Sure, the AI reads your giant prompt fast enough. But the moment it starts writing its response, everything comes to an absolute halt. To fix this lag, we must look at how graphics cards handle data during a chat.
Phase 1: Reading vs. Phase 2: Writing
Running an AI happens in two main steps: the prefill phase (reading) and the decoding phase (writing).
When the AI reads your prompt, your graphics card processes the entire input prompt at once. The model computes attention scores for all tokens against all other tokens in a single step. This requires dense matrix multiplications which keeps the graphics card running at full blast.
But writing the response is a totally different story. The model generates text autoregressively, writing one single token at a time. To generate the next word in a sequence, it has to calculate the attention score of the token by comparing its similarity against the Key-Value (KV) cache that houses all preceding tokens.
If your prompt is 100,000 words long, that cache contains a massive mountain of data. For every single new word the AI spits out, the graphics card must drag the entire mountain from High-Bandwidth Memory (HBM) and into its local Static Random-Access Memory.
So basically, the model only has to do a tiny bit of math for one word, but in order to do that math, it needs to move a large volume of data. The Tensor Cores end up sitting idle, waiting for the memory to finish copying the KV-cache. The decoding phase isn’t slow because of heavy compute, it’s slow because of the data transfer.
Why Traditional FlashAttention Struggles with Long Prompts
Engineers invented a tool called FlashAttention to help speed things up. It works well for reading prompts because it stops the chip from making messy, giant data spreadsheets.
Traditional FlashAttention splits its work across the number of concurrent requests it is handling at once, and the number of "heads" (think of them as different lenses) the AI uses to look at data.
Modern graphics cards are packed with well over a hundred independent compute units, known as Streaming Multiprocessors (SMs). But massive frontier models are usually split across an eight-GPU cluster using Tensor Parallelism. This shards the model's query heads, leaving each individual card holding a fraction, maybe 8 or 16. When you are only generating a single response (a batch size of 1), the engine only fires up one task per head. So, only 8 or 16 of your hundred-plus SMs are actually doing anything. Over 90% of the very expensive chip is just sitting there idling, while a small handful of units get bogged down trying to pull the massive KV-cache out of storage.
The Fix: Splitting the Timeline
Flash-Decoding addresses this waste by introducing a third axis of parallelization: the sequence dimension.
Instead of assigning the entire KV-cache to one single unit, Flash-Decoding splits it into smaller blocks. If a request has a length of 100,000 tokens, the engine can divide this into ten blocks of 10,000 tokens each. Then, the scheduler hands those blocks out to different workspace engines across the GPU simultaneously.
Every unit tackles its own chunk at the same time. They load a fraction of the total KV cache, do their calculations for local attention scores, and track the computed values. Once all of the SMs finish their computations, the GPU runs a quick cleanup step. It assigns an independent thread block to each attention head. These blocks collect all of the tracked attention scores, rescales them to ensure accuracy, and combines them into the final output token representation.
By breaking up the sequence, Flash-Decoding ensures the whole GPU is being utilized. No more idle processors. The memory bottleneck is reduced by splitting the workload across the entire Graphics Card. The result is fast generation speeds, even when handling a textbook-sized prompt.