I think the next major LLM breakthrough may not be a bigger model. It may be a model that fits where useful work actually happens.
We are spending too much time looking at parameter counts. Bigger models are impressive, and I do not want to dismiss that. I wrote last week about the case for routing requests to the right model, not always the biggest one, and the underlying logic is the same one I want to push further here. Every model eventually hits physical limits, and physical limits do not negotiate with parameter counts.
The Three Walls #
If you watch a model being served in production, you start to notice that there are really only three things that can slow it down, and they show up in a predictable order.
The first is computecomputeThe processing capacity of the GPU during the prefill phase, where every input token must be processed before the model can begin generating output. . The prefill phase, where the GPU has to chew through every token of the input before generation can start, is bound by how fast the hardware can process attention and feed-forward operations across the entire prompt. A huge prompt means the user waits, even before the first token of the answer appears. The faster the prefill, the more interactive the system feels.
The second is I/O, and specifically memory bandwidth. Generation happens one token at a time, and at each step the model has to read its weights and shuffle the KV cache around. That is not a single bottleneck. It is two bottlenecks stacked. Memory bandwidth on the HBM, and interconnect bandwidth between GPUs when the model does not fit on one. The cost of generating a long answer is not the cost of the answer. It is the cost of streaming the answer through hardware that has a finite bandwidth budget.
The third is memory. A 1M-token context window sounds great on a slide. The same window becomes a different conversation once you price the KV cache for a million tokens at inference time, with every new token added to the cache and every old one still sitting there. Context is not free. Short-term memory has a real infrastructure price, and that price changes which products are even buildable on top of a given model.
These three walls do not argue against large models. They argue against large models being the only thing you build, and against large models being the default for every request.
What Useful Compression Looks Like #
I watch the other side of the race closely, the side that takes capable models and makes them smaller, faster, cheaper, and easier to run. That is the side that decides whether a model becomes a product or stays a research result.
NVIDIA’s Nemotron 3 Puzzle is a good example of what serious compression work looks like in 2026. They took a 120B (active 12B) mixture-of-experts model and compressed it into a 75.3B model using an approach they call Iterative Puzzle, which combines pruningpruningThe practice of removing parameters from an existing neural network to reduce its size and computational cost while preserving as much capability as possible. , distillationdistillationThe process of transferring knowledge from a large pre-trained teacher model to a smaller student model, so the smaller model learns to mimic the behavior of the larger one. , quantizationquantizationReducing the numerical precision of a model's weights and activations to lower memory and compute requirements, often with minimal quality loss. , reinforcement learning from human feedback, and multi-token prediction. Wikipedia+3 4 sources 2 Pruning (artificial neural network)Wikipedia 3 Knowledge distillationWikipedia 4 Large language modelWikipedia 5 Reinforcement learning from human feedbackWikipedia
That is not a single trick. That is a stack of techniques, each one losing a little quality and gaining a lot of efficiency, applied iteratively until the budget makes sense.
The numbers NVIDIA reports are the interesting part, not because numbers settle arguments but because they show what becomes possible when compression is taken seriously.
On an 8xB200 node, under matched interactive-serving constraints, they report roughly 2x higher server throughput. That is not a small number. That is the difference between “a model you can serve to a small team” and “a model you can serve to a small company.”
For ultra-long context on a single H100, the more striking number is this. They report moving from one concurrent 1M-token request to eight. Eight times the concurrency, on the same hardware, at the same context length. That is the same KV cache pressure I was describing earlier, except now the model leaves enough memory headroom to actually use that context.
What This Is Not Just a Smaller Model #
A naive reading of “75.3B instead of 120B” is that you got less model for less money. That is the wrong reading.
What you actually got is fewer GPUs required to serve a given workload, lower per-request cost, more concurrent users on the same hardware, and crucially, enough VRAM left over to make the context window useful. That remaining memory budget is where many real products either become possible or fail. If your model eats all the VRAM, your context window is a theoretical feature. If your model leaves headroom, your context window is a product surface.
The same economics show up at every scale. I have been running a personal agent that touches a billion tokens of inference at small scale, and the lesson there is the same one Nemotron 3 Puzzle demonstrates at large scale. The economics are not a footnote. They are the design input. The infrastructure, the electricity, the network, the memory topology, the KV cache pressure – none of that is decoration around the model. It is the model, in the sense that it decides what the model can actually do in the world.
What We Should Be Asking #
The question is not only “how intelligent is this model?” We should also be asking how useful it is to society, how much each request costs, how much context we can afford, and how many GPUs we need before the deployment stops being reasonable. A 1000B-parameter model may be very capable, but if it needs a datacenter to serve a practical number of users, it is not a useful solution for most companies. It is a great research result, but not necessarily a deployable product.
That is the same argument I made in the routing piece from earlier this week, just with the compression dial turned up. Routing asks “which model should handle this request?” Compression asks “which model can we afford to run at all?” Both questions are downstream of the same economics, and both answers are downstream of the same thing I keep watching – useful intelligence per GPU, per watt, and per dollar.
We need frontier models, and we should keep pushing the limits of capability. But the future will not belong only to the company with the largest model. It will also belong to the teams that deliver more useful intelligence per GPU, per watt, and per dollar.
That is the race worth watching.