The Hidden Bottlenecks in LLM Inference: Why TFLOPs and VRAM Don’t Determine Performance

The Hidden Bottlenecks in LLM Inference

1/24/2026

Why TFLOPs and VRAM Are the Least Interesting Parts of Production AI

Introduction: The GPU Fallacy

When organizations plan large-scale LLM inference, the conversation almost always starts with hardware:

How many GPUs?
How much VRAM?
How many TFLOPs?
What’s the max tokens per second?

Those numbers matter — but they are not where most production latency or cost comes from.

This fixation on raw compute is a textbook example of what I’ve previously called the AI Illusion: the belief that advanced infrastructure automatically produces outcomes. In reality, inference performance is determined far more by the system's behavior than by GPU specs.

This article breaks down the hidden bottlenecks that dominate real-world LLM inference and explains why architects who only model TFLOPs and VRAM are consistently surprised in production.

This post breaks down the hidden bottlenecks in LLM inference in detail.
If you want the architectural overview, watch the video above.
If you want the deep dive, keep reading below.

Inference Is Not a Single Step — It’s a Pipeline

Most mental models of inference look like this:
Prompt → GPU → Response

Production inference actually looks more like:

Inference Pipeline (What Actually Happens)


Client
 → Tokenization (CPU)
 → Request queue
 → Scheduler & batching
 → KV-cache lookup
 → Network hops
 → GPU execution
 → KV-cache update
 → Detokenization
 → Token streaming response

Only one of those steps is dominated by GPU math.

Everything else is where latency, jitter, and cost quietly accumulate.

1. Tokenization: The First Invisible Latency Tax

Tokenization is almost always:

CPU-bound
Poorly parallelized
Repeated on every request

Why this matters

Long prompts can add tens of milliseconds before inference even starts
Multi-tenant systems often serialize tokenization under load
Tokenization throughput frequently becomes the first scaling wall

Common architectural mistake: Tokenization is rarely included in latency budgets or capacity models. Teams benchmark GPU throughput while quietly ignoring the CPU path that feeds it.

This is why many inference stacks show excellent GPU utilization but still miss latency SLAs.

2. KV-Cache: Where VRAM Actually Goes

KV-cache is the single most misunderstood component of inference.
It:

Grows linearly with sequence length × layers × heads
Consumes VRAM faster than most architects expect
Determines maximum concurrency more than the model size does

What breaks in production

Fragmentation reduces usable VRAM
Cache eviction introduces unpredictable latency spikes
High concurrency forces tradeoffs between batch size and context length

In many real deployments, KV-cache memory exceeds model weights.

Architectural illusion“Model fits in memory” does not mean “system scales.”

3. Networking: Death by a Thousand Microseconds

Inference traffic is fundamentally different from training traffic:

East-west, not north-south
Bursty, not steady
Latency-sensitive, not throughput-optimized

Hidden costs

Token streaming dramatically increases packet counts
Multi-GPU and multi-node inference introduce synchronization delays
CPU↔GPU↔NIC handoffs add jitter under load

Common mistake
Designing inference networks like training fabrics — or worse, like general IT traffic — guarantees inconsistent tail latency.

4. Contention: The Bottleneck Nobody Benchmarks

Contention exists everywhere in inference systems:

CPU cores handling tokenization and scheduling
PCIe lanes shared across accelerators
Memory bandwidth during concurrent KV-cache access
Network queues under burst traffic

Why benchmarks lie
Most benchmarks:

Run in isolation
Avoid multi-tenant contention
Measure averages instead of P95/P99

This explains why proofs-of-concept look great while production deployments feel “mysteriously slow.”
This pattern shows up repeatedly when organizations move from discovery to AI outcomes — the exact transition where architectural shortcuts are exposed.

5. Batching Policies: Throughput vs. User Experience

Batching improves GPU efficiency — but at a cost.

Larger batches increase time-to-first-token (TTFT)
Interactive workloads suffer
Tail latency becomes unpredictable

The real tradeoff

Optimize for throughput → unhappy users
Optimize for responsiveness → idle GPUs

Most teams optimize for averages and are shocked when P99 latency explodes.

This is a classic Amplification Trap: small inefficiencies scale linearly with usage and rapidly dominate cost.

6. Runtime Choices: Same Model, Radically Different Results

Inference behavior varies wildly depending on the runtime stack.
Differences emerge in:

Scheduler design
KV-cache layout
Tensor parallelism strategy
Memory allocation behavior
Token streaming architecture

Two teams can deploy the same model on the same GPUs and see 2–5× differences in latency and cost.

Treating the inference runtime as an “implementation detail” is one of the most expensive mistakes teams make.

The Real Bottleneck Stack (What Architects Should Model)

Instead of starting with GPUs, architects should model:

Prompt length distributions
Tokenization throughput per CPU core
KV-cache growth vs. concurrency
Queueing and scheduling behavior
Network topology and jitter
Batching policies aligned to SLAs
Runtime memory behavior

Only after this does TFLOPs become relevant.

This aligns directly with the failure patterns outlined in Why AI Projects Fail – The 5 Pillars: inference failures are rarely about models alone. They are architectural, operational, and economic failures.

Why This Keeps Happening

Because:

Hardware specs are easy to reason about
GPUs are visible on budgets
Software bottlenecks don’t show up on invoices

The result is a familiar pattern:

Plenty of GPUs, poor latency, and rising inference costs with no obvious explanation.

Inference Is a Systems Problem

LLM inference is not:

A model problem
A GPU problem
Even strictly an ML problem

It is a distributed systems problem with tight latency constraints and brutal cost sensitivity.

This is why inference architecture fits naturally into an AI Factory mindset: inference must be designed, measured, governed, and optimized as a production system — not bolted onto general infrastructure after the fact.

If you only size for TFLOPs and VRAM, you’re optimizing the least interesting part of the stack.

Your Definitive Source for Actionable Insights on Cloud, Virtualization & Modern Enterprise IT