[ad_1]
As agentic AI strikes from experiments to actual manufacturing workloads, a quiet however critical infrastructure drawback is coming into focus: reminiscence. Not compute. Not fashions. Reminiscence.
Underneath the hood, at this time’s GPUs merely don’t have sufficient area to carry the Key-Worth (KV) caches that fashionable, long-running AI brokers depend upon to keep up context. The result’s loads of invisible waste — GPUs redoing work they’ve already carried out, cloud prices climbing, and efficiency taking a success. It’s an issue that’s already displaying up in manufacturing environments, even when most individuals haven’t named it but.
At a latest cease on the VentureBeat AI Affect Collection, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the business’s rising “reminiscence wall,” and why it’s changing into one of many largest blockers to scaling really stateful agentic AI — programs that may keep in mind and construct on context over time. The dialog didn’t simply diagnose the problem; it laid out a brand new means to consider reminiscence fully, by way of an method WEKA calls token warehousing.
The GPU reminiscence drawback
“Once we're trying on the infrastructure of inferencing, it isn’t a GPU cycles problem. It's principally a GPU reminiscence drawback,” mentioned Ben-David.
The basis of the problem comes right down to how transformer fashions work. To generate responses, they depend on KV caches that retailer contextual data for each token in a dialog. The longer the context window, the extra reminiscence these caches eat, and it provides up quick. A single 100,000-token sequence can require roughly 40GB of GPU reminiscence, famous Ben-David.
That wouldn’t be an issue if GPUs had limitless reminiscence. However they don’t. Even probably the most superior GPUs high out at round 288GB of high-bandwidth reminiscence (HBM), and that area additionally has to carry the mannequin itself.
In real-world, multi-tenant inference environments, this turns into painful rapidly. Workloads like code growth or processing tax returns rely closely on KV-cache for context.
“If I'm loading three or 4 100,000-token PDFs right into a mannequin, that's it — I've exhausted the KV cache capability on HBM,” mentioned Ben-David. That is what’s often known as the reminiscence wall. “All of the sudden, what the inference setting is pressured to do is drop information," he added.
Meaning GPUs are always throwing away context they’ll quickly want once more, stopping brokers from being stateful and sustaining conversations and context over time
The hidden inference tax
“We always see GPUs in inference environments recalculating issues they already did,” Ben-David mentioned. Techniques prefill the KV cache, begin decoding, then run out of area and evict earlier information. When that context is required once more, the entire course of repeats — prefill, decode, prefill once more. At scale, that’s an unlimited quantity of wasted work. It additionally means wasted vitality, added latency, and degraded person expertise — all whereas margins get squeezed.
That GPU recalculation waste exhibits up instantly on the steadiness sheet. Organizations can endure practically 40% overhead simply from redundant prefill cycles That is creating ripple results within the inference market.
“In the event you have a look at the pricing of enormous mannequin suppliers like Anthropic and OpenAI, they’re truly educating customers to construction their prompts in ways in which enhance the chance of hitting the identical GPU that has their KV cache saved,” mentioned Ben-David. “In the event you hit that GPU, the system can skip the prefill section and begin decoding instantly, which lets them generate extra tokens effectively.”
However this nonetheless doesn't remedy the underlying infrastructure drawback of extraordinarily restricted GPU reminiscence capability.
Fixing for stateful AI
“How do you climb over that reminiscence wall? How do you surpass it? That's the important thing for contemporary, cost- efficient inferencing,” Ben-David mentioned. “We see a number of firms attempting to unravel that in numerous methods.”
Some organizations are deploying new linear fashions that attempt to create smaller KV caches. Others are targeted on tackling cache effectivity.
“To be extra environment friendly, firms are utilizing environments that calculate the KV cache on one GPU after which attempt to copy it from GPU reminiscence or use a neighborhood setting for that,” Ben-David defined. “However how do you try this at scale in an economical method that doesn't pressure your reminiscence and doesn't pressure your networking? That's one thing that WEKA helps our prospects with.”
Merely throwing extra GPUs on the drawback doesn’t remedy the AI reminiscence barrier. “There are some issues that you simply can not throw sufficient cash at to unravel," Ben-David mentioned.
Augmented reminiscence and token warehousing, defined
WEKA’s reply is what it calls augmented reminiscence and token warehousing — a strategy to rethink the place and the way KV cache information lives. As a substitute of forcing every part to suit inside GPU reminiscence, WEKA’s Augmented Reminiscence Grid extends the KV cache into a quick, shared “warehouse” inside its NeuralMesh structure.
In follow, this turns reminiscence from a tough constraint right into a scalable useful resource — with out including inference latency. WEKA says prospects see KV cache hit charges bounce to 96–99% for agentic workloads, together with effectivity positive aspects of as much as 4.2x extra tokens produced per GPU.
Ben-David put it merely: "Think about that you’ve got 100 GPUs producing a certain quantity of tokens. Now think about that these hundred GPUs are working as in the event that they're 420 GPUs."
For giant inference suppliers, the outcome isn’t simply higher efficiency — it interprets on to actual financial impression.
“Simply by including that accelerated KV cache layer, we're taking a look at some use circumstances the place the financial savings quantity can be thousands and thousands of {dollars} per day,” mentioned Ben-David
This effectivity multiplier additionally opens up new strategic choices for companies. Platform groups can design stateful brokers with out worrying about blowing up reminiscence budgets. Service suppliers can provide pricing tiers primarily based on persistent context, with cached inference delivered at dramatically decrease price.
What comes subsequent
NVIDIA tasks a 100x enhance in inference demand as agentic AI turns into the dominant workload. That stress is already trickling down from hyperscalers to on a regular basis enterprise deployments— this isn’t only a “huge tech” drawback anymore.
As enterprises transfer from proofs of idea into actual manufacturing programs, reminiscence persistence is changing into a core infrastructure concern. Organizations that deal with it as an architectural precedence somewhat than an afterthought will achieve a transparent benefit in each price and efficiency.
The reminiscence wall shouldn’t be one thing organizations can merely outspend to beat. As agentic AI scales, it is likely one of the first AI infrastructure limits that forces a deeper rethink, and as Ben-David’s insights made clear, reminiscence may additionally be the place the subsequent wave of aggressive differentiation begins.
[ad_2]
