Scaling AI Inference with Nutanix Enterprise AI: Sharding, Instances, KV Cache and Agent Gateway
When discussing AI inference infrastructure, the word “scaling” can mean several very different things. We might need more GPUs because a model is too large for a single GPU, more copies of a model because concurrency is increasing, or better routing between those copies because repeated context is consuming unnecessary compute. Eventually, we may also reach the point where we want the AI service to grow beyond the capacity of a single Nutanix Enterprise AI (NAI) environment.
Nutanix Enterprise AI provides different mechanisms for addressing each of these requirements. The easiest way to think about them is as three levels of scaling: Scale Up, Scale Out and Scale Beyond. Scale Up changes the GPU resources available to an individual model instance. Scale Out increases the number of model instances behind a local endpoint. Scale Beyond moves the scaling boundary above the individual NAI environment by using Nutanix Agent Gateway and additional NAI inference capacity.
The distinction is important because these approaches have different operational characteristics. In particular, increasing the number of local model instances is very different from changing the number of GPUs used to shard each instance.
Scale Up: Local Endpoint Sharding
The first approach is to increase the number of GPUs used by an individual model instance. A sufficiently large model may simply not fit within the memory available on one GPU, or the required inference characteristics may justify distributing the model across multiple GPUs. In this case, several GPUs collectively provide the resources required to run a single instance of the model.
For example, an endpoint might contain one model instance using four GPUs:
Model Instance → GPU 1 + GPU 2 + GPU 3 + GPU 4
This is primarily a model execution decision. Those four GPUs are not four independent copies of the model; they are collectively supporting the same model instance. This is an important distinction when sizing an inference platform because adding GPUs through sharding does not necessarily provide the same concurrency benefit as creating additional independent model instances.
There is also an important operational consideration. The GPU topology is defined when the local endpoint is deployed. If an endpoint was originally created with two GPUs per instance and we subsequently decide that each instance should use four GPUs, we cannot simply add another two GPUs to the existing endpoint. The endpoint must be deleted and recreated with the new GPU topology.
This makes sharding a form of Scale Up. It is the appropriate mechanism when the model itself requires additional GPU resources, but changing that topology is a more significant operation than increasing the number of instances behind an existing endpoint.
Scale Out: Multiple Local Endpoint Instances
The second mechanism is to increase the number of model instances behind the local endpoint. Instead of making an individual model instance larger, we create additional copies of that instance so that more inference requests can be processed concurrently.
Imagine that our model requires two GPUs. We might initially deploy one instance using two GPUs. As demand grows, we could increase this to four instances, with each instance retaining the same two-GPU topology. The result would be eight GPUs in total, but they would be arranged as four independent two-GPU model-serving instances rather than one eight-GPU model instance.
Conceptually, the endpoint would look like this:
Local Endpoint
Instance 1 → GPU 1 + GPU 2
Instance 2 → GPU 3 + GPU 4
Instance 3 → GPU 5 + GPU 6
Instance 4 → GPU 7 + GPU 8
The important operational difference is that additional instances can be added to the local endpoint without deleting and recreating it. The GPU topology of each instance remains unchanged; we are simply increasing the number of workers available to process inference requests.
This makes multiple instances the natural way to scale when the model already runs correctly on the allocated GPUs but demand is increasing. Sharding answers the question, “How much GPU resource does each copy of my model require?” Instance scaling answers a different question: “How many copies of that model do I need to service the workload?”
That distinction becomes particularly important in production. An endpoint might begin with two GPUs per instance and two instances, consuming four GPUs in total. If demand increases, it could grow to four or eight instances without changing the underlying two-GPU model topology. If we subsequently determine that every instance needs four GPUs instead of two, however, we have crossed back into Scale Up territory and the endpoint needs to be recreated with the new sharding configuration.
| Change | Scaling Type | Endpoint Recreation? |
| 1 GPU → 2 GPUs per instance | Scale Up / Sharding | Yes |
| 1 instance → 4 instances | Scale Out / Instances | No |
| 2 GPUs × 2 instances → 2 GPUs × 4 instances | Scale Out | No |
| 2 GPUs × 4 instances → 4 GPUs × 4 instances | Scale Up | Yes |
This difference gives NAI two useful local scaling characteristics. We can plan the GPU topology around the requirements of the model while retaining the ability to increase the number of model-serving instances as demand grows.
KV Cache Aware Routing: Making Scale Out Smarter
Once an endpoint contains multiple model instances, another problem appears: which instance should receive each request? Traditional load balancing might distribute requests using round robin, least connections or another infrastructure-oriented algorithm. LLM inference introduces another consideration because the instances may contain different useful information in their KV caches.
During inference, key and value representations for previously processed tokens can be retained in the KV cache. This means the model does not necessarily need to repeat all of the computational work associated with previously processed context. When prompts are large, or when substantial parts of the context are repeated between requests, reuse of that cached information can have a meaningful effect on inference efficiency.
Consider a coding assistant processing a large repository context. The first request is handled by Instance 1, which now has useful cached information associated with that request or its common prompt prefix. A subsequent request from the same workload arrives, but a conventional load balancer decides that Instance 2 is currently less busy. From a traditional infrastructure perspective that may be a perfectly sensible decision, but from an inference perspective it could be inefficient because Instance 1 may already contain useful cached context.
KV Cache Aware Routing allows cache information to become part of the routing decision. Instead of only asking which instance has capacity, the platform can also consider which instance already has useful cached context for the incoming request. This makes the routing layer aware of the characteristics of LLM inference rather than treating each request as an interchangeable HTTP transaction.
The relationship with local instance scaling is important. With a single model instance there is no meaningful choice about where a request should be sent. As we scale to two, four, eight or more instances, request placement becomes increasingly important. KV Cache Aware Routing therefore becomes more relevant as we Scale Out.
Where KV Cache Aware Routing Matters Most
The value of KV Cache Aware Routing will not be equal across every workload. A short, stateless prompt with little repeated context may have relatively limited opportunity for cache reuse. Long-running conversations, coding assistants, RAG applications, large system prompts and agentic workloads present a very different profile because significant portions of their context may be repeated between requests.
Agentic workloads are particularly interesting. An agent may repeatedly submit a large system prompt together with tool definitions, conversation history, retrieved information and instructions governing how it should operate. Coding agents may additionally include repository context, coding standards and information about files that have already been examined. Sending these requests indiscriminately between model instances can result in GPUs repeatedly processing context that has already been processed elsewhere.
This is why I think of KV Cache Aware Routing as inference-aware routing rather than simply load balancing. It does not add GPUs or create additional model instances. Instead, it attempts to make better use of the inference capacity that is already available. As context windows grow and agentic applications generate increasingly complex request patterns, that distinction becomes more significant.
Scale Beyond: Nutanix Agent Gateway
There is eventually another scaling boundary to consider. We may be able to continue adding model instances within a local NAI environment, but there are good reasons why we may not want the capacity of one environment to become the ultimate boundary of the AI service. The cluster may be approaching its available GPU capacity, another cluster may have unused resources, or we may deliberately want inference capacity distributed between physical locations or failure domains.
This is where Nutanix Agent Gateway changes the architecture. Instead of applications connecting directly to an individual model endpoint, Agent Gateway can provide a Unified Endpoint in front of multiple inference targets. Additional NAI environments can then contribute inference capacity behind that abstraction.
A simplified architecture could look like this:
Applications and Agents → Agent Gateway Unified Endpoint → NAI A + NAI B + NAI C
Each of those NAI environments can then have its own local endpoint configuration, and each local endpoint can contain multiple model instances. We have therefore introduced another scaling layer above the local endpoint itself.
I describe this as Scale Beyond, because we are no longer asking how large a single endpoint should become. Instead, we are asking how much inference capacity should exist behind the AI service presented to applications.
Agent Gateway and Remote NAI
Consider an NAI environment containing an endpoint with four instances, each using two GPUs. The endpoint therefore consumes eight GPUs. If local demand increases and sufficient GPU capacity exists in the environment, we can simply add more model instances. That remains the simplest form of horizontal scaling.
Eventually, however, we may decide that another NAI environment should provide the additional capacity. Perhaps the original cluster is full, another Nutanix cluster has available GPUs, or we want the service to span multiple sites. Rather than continually expanding the original environment, another NAI deployment can provide additional model-serving capacity behind Agent Gateway.
We might therefore have:
Agent Gateway Unified Endpoint
NAI A → 4 instances × 2 GPUs
NAI B → 4 instances × 2 GPUs
The application continues to consume the same logical service while the inference estate underneath it has expanded. If further capacity is required, another NAI environment can be introduced without requiring applications to be redesigned around the underlying physical topology.
This is an important difference between local instance scaling and Agent Gateway. Local instance scaling grows the capacity of an endpoint within an NAI environment. Agent Gateway allows us to grow the service across inference targets and NAI environments.
Agent Gateway and Multiple Local Model Instances
Agent Gateway and local instance scaling should therefore not be viewed as competing approaches. They operate at different layers and can be used together.
A production environment might contain an Agent Gateway Unified Endpoint in front of three NAI environments. Each NAI environment could have an endpoint running four instances of the same model, and each of those instances could itself use two GPUs. KV Cache Aware Routing can then improve request placement between the local instances, while Agent Gateway provides the higher-level abstraction through which applications consume the AI service.
That gives us three distinct scaling layers:
| Layer | Mechanism | What It Solves |
| Scale Up | GPU sharding | Model size and per-instance GPU requirements |
| Scale Out | Additional local instances | Concurrency and aggregate inference throughput |
| Scale Beyond | Agent Gateway + additional NAI capacity | Service-level and infrastructure-level scaling |
KV Cache Aware Routing sits alongside Scale Out because it helps improve how efficiently multiple local instances are used.
This layered model is important because a production AI platform may use all of these techniques simultaneously. A large model might require two or four GPUs per instance, each NAI endpoint might contain multiple instances to handle concurrency, and several NAI environments might then sit behind Agent Gateway to provide greater aggregate capacity or infrastructure resiliency.
A Practical Growth Example
Imagine an organisation deploying an internal coding assistant for an initial population of 100 developers. The selected model requires two GPUs per instance, and testing suggests that two instances provide sufficient initial capacity. The local endpoint is therefore consuming four GPUs.
As adoption increases, latency and concurrency begin to become a concern. The model itself still runs comfortably on two GPUs, so there is no reason to change its sharding configuration. Instead, additional local instances can be added. Moving from two instances to four increases the total GPU consumption from four to eight GPUs while leaving the model topology unchanged.
At this point KV Cache Aware Routing becomes increasingly valuable. Coding assistants frequently reuse large amounts of context, so intelligently directing requests towards instances that contain useful cached information can help improve the efficiency of the expanded inference estate.
Now imagine adoption grows from 100 developers to several thousand, with coding agents increasingly operating autonomously. The original NAI environment may no longer be the place where we want to concentrate all of that inference capacity. A second NAI environment can be introduced and its inference capacity exposed through Agent Gateway alongside the original environment.
The progression has therefore been straightforward. We started by determining how many GPUs each model instance required. We then added model instances as concurrency increased. We used KV Cache Aware Routing to improve the efficiency of those instances. Finally, Agent Gateway allowed the service to expand beyond the capacity or architectural boundary of the original NAI environment.
Why Agentic AI Makes This More Important
These distinctions become more important as enterprise AI moves from conventional chat interfaces towards agents. With a traditional application, user numbers and request rates can provide at least a reasonable starting point for infrastructure sizing. Agentic workloads make the relationship between users and inference demand considerably less predictable.
A single human request might cause an agent to make dozens of model calls as it reasons through a task, invokes tools, evaluates results, retries operations and potentially delegates work to other agents. The infrastructure is no longer servicing only the interactions directly generated by humans; it is servicing a growing amount of machine-generated inference demand as well.
At the same time, those agent interactions often contain large amounts of repeated context. This creates a requirement not only for more inference capacity but also for better utilisation of that capacity. Local instance scaling provides additional concurrency, KV Cache Aware Routing can improve the efficiency of request placement, and Agent Gateway provides the abstraction through which the overall AI service can continue to expand.
From GPU Scaling to AI Service Scaling
For me, this is the more useful way to think about scaling Nutanix Enterprise AI. The conversation begins with the GPU and the question of how much resource an individual model instance requires. It then moves to the endpoint and how many copies of that model are necessary to service the workload. Once multiple instances exist, the efficiency of routing between them becomes important. Eventually, the conversation moves beyond the endpoint entirely and becomes a question of how much inference capacity should sit behind the AI service.
That gives us a relatively simple progression: Scale Up → Scale Out → Scale Beyond. Sharding provides model scale, multiple local instances provide throughput scale, KV Cache Aware Routing helps improve inference efficiency as we scale out, and Agent Gateway provides service-level scale and abstraction across a wider inference estate.
This is an important evolution in how we think about private AI infrastructure. The objective should not be for every application developer to understand how many GPUs a model consumes, how many instances are running or which NAI environment currently has available capacity. Applications and agents should consume an AI service, while the infrastructure underneath that service is free to evolve as requirements change.
That is where the combination of Nutanix Enterprise AI, multiple model instances, KV Cache Aware Routing and Nutanix Agent Gateway becomes particularly interesting. It moves the conversation beyond simply running models on GPUs and towards operating a scalable enterprise inference platform.