Orchestrating Kubernetes AI Inference Workloads with NVIDIA Grove — From DRA GA to KAI Scheduler Integration

Why Existing Kubernetes Alone Falls Short for AI Inference Workloads

In March 2026, at KubeCon Europe 2026 in Amsterdam, NVIDIA officially announced the open-source project Grove. Grove is a Kubernetes API for declaratively defining and orchestrating complex AI inference systems in Kubernetes. The era of simply spinning up a single Pod is over.

Modern LLM inference architectures are evolving toward Disaggregated Inference patterns. The Prefill and Decode stages are separated, KV-Cache is shared, and each component must scale independently. The traditional Kubernetes combination of Deployment + HPA cannot properly manage such composite workloads.

Core Problem: When you scale out Prefill workers, you need to proportionally increase Decode capacity, and when adding new service instances, you must maintain the correct ratio of each component. Pod-level autoscaling cannot express these dependencies.

Grove was designed as a core component of the NVIDIA Dynamo framework to solve this problem, developed with integration into the CNCF ecosystem as a prerequisite. At the same KubeCon, NVIDIA also announced the historic decision to donate their GPU DRA driver to the CNCF.

Grove Architecture: 3-Tier CRD Model

Grove abstracts AI inference workloads into three Custom Resource Definition (CRD) layers. Each layer separates different levels of concerns while ensuring the entire system is managed as a single declarative resource.

[Grove 3-Tier CRD Architecture]

┌─────────────────────────────────────────────────────────────────┐
│ PodCliqueSet │
│ (Full inference service definition: startup order, scaling policies, Gang scheduling) │
│ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ │
│ │ PodCliqueScalingGroup │ │ PodCliqueScalingGroup │ │
│ │ (Groups that scale │ │ (Independent scale │ │
│ │ together) │ │ group) │ │
│ │ │ │ │ │
│ │ ┌─────────┐ ┌────────┐ │ │ ┌──────────────────┐ │ │
│ │ │PodClique│ │PodCliq.│ │ │ │ PodClique │ │ │
│ │ │(Prefill │ │(Prefill│ │ │ │ (Decode │ │ │
│ │ │ Leader) │ │Worker) │ │ │ │ Workers) │ │ │
│ │ └─────────┘ └────────┘ │ │ └──────────────────┘ │ │
│ └──────────────────────────┘ └──────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

PodClique: Role-Based Pod Groups

PodClique is a group of Pods performing a specific role. Each role – Prefill Leader, Prefill Worker, Decode Leader, Decode Worker – has independent configuration and scaling logic appropriate for its role. Each PodClique can have different GPU requirements, images, and commands.

PodCliqueScalingGroup: Tightly Coupled Scaling

PodCliqueScalingGroup bundles PodCliques that must be scaled together. For example, if 1 Prefill Leader and 4 Prefill Workers constitute a single model instance, they are managed as one ScalingGroup. This prevents inconsistencies where only Workers are scaled without the Leader during scale-out operations.

PodCliqueSet: Full Service Orchestration

PodCliqueSet defines the entire multi-component workload as a single declarative resource. It includes startup order (Workers start after Leader is ready), scaling policies, and Gang scheduling constraints.

In-Depth Analysis of Core Features

1. Multi-Level Auto-Scaling

Grove handles 3-tier autoscaling simultaneously. Independent yet coordinated scaling occurs at the individual component level (adding Prefill Workers during traffic spikes), related component group level (scaling Prefill Leader and Workers together), and entire service instance level (expanding overall capacity).

2. Flexible Gang Scheduling

Traditional Gang scheduling follows an “all-or-nothing” approach. Grove improves this by ensuring minimum viable component combinations while allowing flexible scaling. Prefill and Decode components can scale at different ratios based on workload patterns.

The Grove operator automatically creates PodGang resources when processing PodCliqueSet resources. PodGang encapsulates minimum replica guarantees, network topology preferences, and distribution constraints to pass to the scheduler.

3. Topology-Aware Placement

GPU topology directly impacts performance in AI inference. In NVIDIA GB200 NVL72 systems, placing related Prefill and Decode Pods in the same NVLink domain can significantly reduce KV-Cache transfer latency. Grove automatically performs such topology-aware placement by integrating with KAI Scheduler.

Tip: KAI Scheduler is an open-source subcomponent of NVIDIA Run:ai, recently adopted as a CNCF Sandbox project. It continuously monitors PodGang resources and makes placement decisions considering GPU topology and cluster locality.

4. Role-Aware Orchestration

Prefill Leaders and Workers execute different startup logic. Workers cannot start before the Leader is ready. Grove manages these explicit startup orders and role-specific configurations at the CRD level. When failed components restart, they reconnect correctly at the system level, and during rolling updates, network topology is preserved to maintain low-latency states.

Practical Deployment: Deploying Qwen3 Model with KV-Routing

Let’s walk through setting up an actual Grove environment and deploying a Qwen3 0.6B model using the Disaggregated Inference pattern step by step.

Step 1: Install Prerequisites

# Create namespace
kubectl create namespace vllm-v1-disagg-router

# Install Dynamo CRDs
export NAMESPACE=vllm-v1-disagg-router
export RELEASE_VERSION=0.5.1

helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz --namespace default

# Install Dynamo Operator + Enable Grove
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz
helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz 
  --namespace ${NAMESPACE} --create-namespace 
  --set "grove.enabled=true"

Step 2: Verify CRD Installation

# Check Grove CRDs
kubectl get crd | grep grove

# Expected output:
# podcliques.grove.io
# podcliquescalinggroups.grove.io
# podcliquesets.grove.io
# podgangs.scheduler.grove.io
# podgangsets.grove.io

Step 3: Define DynamoGraphDeployment

Here’s the complete manifest for deploying the Qwen3 0.6B model with Prefill/Decode separation architecture:

# dynamo-grove.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: dynamo-grove
spec:
  services:
    # Frontend: KV-Routing based router
    Frontend:
      dynamoNamespace: vllm-v1-disagg-router
      componentType: frontend
      replicas: 1
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
      envs:
        - name: DYN_ROUTER_MODE
          value: kv   # Enable KV-Cache based routing

    # Decode Worker: Responsible for token generation
    VllmDecodeWorker:
      dynamoNamespace: vllm-v1-disagg-router
      envFromSecret: hf-token-secret
      componentType: worker
      replicas: 2   # Decode needs more instances than Prefill
      resources:
        limits:
          gpu: "1"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
          workingDir: /workspace/components/backends/vllm
          command: [python3, -m, dynamo.vllm]
          args:
            - --model
            - Qwen/Qwen3-0.6B

    # Prefill Worker: Responsible for processing input prompts
    VllmPrefillWorker:
      dynamoNamespace: vllm-v1-disagg-router
      envFromSecret: hf-token-secret
      componentType: worker
      replicas: 1
      resources:
        limits:
          gpu: "1"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
          workingDir: /workspace/components/backends/vllm
          command: [python3, -m, dynamo.vllm]
          args:
            - --model
            - Qwen/Qwen3-0.6B
            - --is-prefill-worker   # Prefill-only mode flag

Step 4: Deploy and Verify

# Deploy
kubectl apply -f dynamo-grove.yaml -n ${NAMESPACE}

# Check Pod status
kubectl get pods -n ${NAMESPACE}
# NAME                                          READY   STATUS    RESTARTS   AGE
# dynamo-grove-frontend-0                       1/1     Running   0          2m
# dynamo-grove-0-vllmdecodeworker-0             1/1     Running   0          2m
# dynamo-grove-0-vllmdecodeworker-1             1/1     Running   0          2m
# dynamo-grove-0-vllmprefillworker-0            1/1     Running   0          2m

# Check PodClique resource status
kubectl get podclique -n ${NAMESPACE}
# NAME                                   READY   REPLICAS
# dynamo-grove-0-vllmdecodeworker        2/2     2
# dynamo-grove-0-vllmprefillworker       1/1     1

# Test API endpoint
kubectl port-forward svc/dynamo-grove-frontend 8000:8000 -n ${NAMESPACE}

# Check model list
curl http://localhost:8000/v1/models

# Test inference request
curl -X POST http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "What is Kubernetes?"}],
    "max_tokens": 512
  }'

Tip: Using KV-Routing mode (DYN_ROUTER_MODE=kv) allows the Frontend to select the optimal Decode Worker based on KV-Cache state. This increases cache hit rates and significantly improves TTFT (Time to First Token).

The Significance of NVIDIA GPU DRA Driver Donation to CNCF

At the same KubeCon Europe 2026, NVIDIA donated their GPU DRA (Dynamic Resource Allocation) driver to the CNCF. This is a historic event where GPU resource management governance transferred from a single vendor to the community.

DRA was promoted to GA (General Availability) in Kubernetes 1.34, fundamentally solving the limitations of the existing Device Plugin model. Key differences include:

With DRA, you can request resources based on detailed GPU attributes as follows:

# DRA ResourceClaim - GPU attribute-based request
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-for-inference
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: >
            device.attributes["gpu.nvidia.com"].productName == "NVIDIA H100" &&
            device.attributes["gpu.nvidia.com"].memory >= 80000000000
    # Priority list: fallback to A100 if H100 unavailable
    - name: gpu-fallback
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: >
            device.attributes["gpu.nvidia.com"].productName == "NVIDIA A100"

Grove + DRA + KAI Scheduler: Integrated Architecture

When these three projects combine, they complete a stack that can fully automate AI inference workloads in Kubernetes:

[Integrated Architecture Flow]

User Request
│
▼
┌──────────────────────────────┐
│ PodCliqueSet (Grove) │ ← Declare entire workload
│ │
│ ┌─── Frontend ────────┐ │
│ │ KV-Router │ │
│ └──────────────────────┘ │
│ │
│ ┌─── Prefill Group ───┐ │
│ │ Leader + Workers │ │ ← PodCliqueScalingGroup
│ └──────────────────────┘ │
│ │
│ ┌─── Decode Group ────┐ │
│ │ Leader + Workers │ │ ← PodCliqueScalingGroup
│ └──────────────────────┘ │
└──────────────────────────────┘
│
▼ Create PodGang
┌──────────────────────────────┐
│ KAI Scheduler │ ← Gang scheduling + topology placement
│ – GPU topology awareness │
│ – NVLink domain optimization │
│ – Resource fragmentation prevention │
└──────────────────────────────┘
│
▼ Process ResourceClaim
┌──────────────────────────────┐
│ DRA (GPU Driver) │ ← Attribute-based GPU allocation
│ – Automatic H100/A100 selection │
│ – MIG profile support │
│ – MPS sharing configuration │
└──────────────────────────────┘

Practical Adoption Considerations

Cluster Requirements

• Kubernetes Version: 1.34+ (DRA GA support required). For 1.33, DRA is beta and requires --feature-gates=DynamicResourceAllocation=true flag.

• GPU Nodes: Nodes with NVIDIA GPUs + NVIDIA DRA driver installation

• Helm 3.12+: Required for Dynamo chart installation

• Scheduler: KAI Scheduler or custom scheduler that understands PodGang resources

Cost Optimization Strategy

Companies adopting the Grove + DRA combination report GPU job queue wait times reduced from 15-45 minutes to 3-10 minutes, monthly GPU costs reduced by 30-40%, and developer experiment iteration speed improved by over 80%.

Caution: Grove is currently in v0.5.x stage, and production stability has not been fully validated. We recommend thorough testing in staging environments before production deployment. Particularly validate PodCliqueSet rolling update strategies and failure recovery scenarios in advance.

Integration with Existing Environments

Using DRA’s Extended Resource Mapping (alpha feature), existing workloads can use DRA-managed resources without modification. Traditional nvidia.com/gpu: 1 resource requests are internally processed through DRA, enabling gradual migration.

Ecosystem Outlook: GPUs Become First-Class Citizens in Kubernetes

Summarizing the trends from KubeCon Europe 2026, GPUs are now establishing themselves as first-class citizens in Kubernetes, equal to containers. NVIDIA’s donation of the DRA driver declares that GPU orchestration is not the domain of a specific vendor but a shared asset of the open-source community.

This ecosystem collaboration involving AWS, Broadcom, Canonical, Google Cloud, Microsoft, Nutanix, Red Hat, and SUSE is expected to accelerate the standardization of cloud-native AI infrastructure. Azure AKS already supports DRA and vGPU integration, and Google GKE officially supports DRA.

This change delivers a clear message to DevOps and MLOps teams. As AI workload complexity increases, Deployment and HPA alone are no longer sufficient. The new orchestration stack consisting of Grove, DRA, and KAI Scheduler represents essential technology that must be understood and mastered as Kubernetes evolves into the operating system for AI infrastructure.

References

• NVIDIA Grove GitHub Repository

• NVIDIA Technical Blog – Streamline Complex AI Inference with Grove

• NVIDIA at KubeCon Europe 2026 Announcements

• Kubernetes v1.34: DRA GA Updates

• NVIDIA DRA Driver for GPUs (CNCF)

• KAI Scheduler – CNCF Sandbox Project

• Grove Deployment Guide – NVIDIA Dynamo Docs

Originally published on ManoIT Tech Blog. Translated from Korean.