NVIDIA CUDA 13.4 and KAI Scheduler Bring Memory-Enforced Fractional GPU Sharing to Kubernetes

NVIDIA has introduced application-transparent, driver-enforced GPU memory boundaries in CUDA 13.4 alongside the open-source GPU Fractioning Operator and KAI Scheduler v0.18.0, resolving one of the longest-standing reliability hazards in multi-tenant Kubernetes AI clusters. For the first time, platform engineers can partition a single physical GPU across multiple containerized workloads using dynamic software fractions—across both Multi-Process Service (sm-sharing) and a new time-sharing execution mode—while guaranteeing that a memory-hungry pod cannot exhaust device VRAM and crash neighboring containers.

Historically, infrastructure teams running lightweight inference microservices, embedding models, or interactive Jupyter notebooks on high-memory data center GPUs faced a rigid architectural trade-off. Hardware Multi-Instance GPU (MIG) provides strict hardware memory isolation, but it supports only coarse, pre-defined slice ratios (such as 1/7th or 1/2nd of an H100/B200) and requires draining workloads to reconfigure partitions. Conversely, software-based Kubernetes GPU time-slicing allows flexible fractional scheduling, but prior to CUDA 13.4 it lacked native driver-level VRAM isolation: any single container could allocate 100% of the physical GPU memory, triggering immediate Out-Of-Memory (OOM) kernel panics across every co-located pod.

CUDA 13.4 closes that gap at the driver layer. By intercepting device memory allocations transparently beneath PyTorch, vLLM, or TensorRT runtimes, the updated driver enforces per-container VRAM ceilings without requiring code changes or custom allocator wrappers inside user containers.

To operationalize these driver capabilities in cloud-native environments, NVIDIA and the KAI Scheduler community released the open-source GPU Fractioning Operator together with native NvFractions support in KAI Scheduler v0.18.0. Cluster administrators can now declare exact megabyte limits, burstable VRAM ranges, or proportional GPU slices directly through standard Kubernetes Pod annotations while receiving isolated per-fraction telemetry in NVIDIA DCGM.

Fast Facts: NVIDIA CUDA 13.4 Memory-Enforced Kubernetes GPU Sharing

• Core Release Components: Driver-level memory boundary enforcement in NVIDIA CUDA 13.4 paired with the open-source GPU Fractioning Operator and KAI Scheduler v0.18.0 (NvFractions mode), detailed in NVIDIA’s technical architecture blog post.
• Supported Execution Modes: Enforces strict VRAM boundaries across both spatial Multi-Process Service (sm-sharing) and temporal context switching via a new time-sharing mode.
• Three Declarative Memory Policies: Supports exact fixed VRAM limits (out-of-bounds allocations fail cleanly inside the offending pod), guaranteed minimum with burstable maximum into unused device memory, and portion-based fractional slices.
• Application Transparency: Requires zero modifications to container images, CUDA kernels, or deep learning frameworks (PyTorch, JAX, TensorRT-LLM).
• Per-Fraction Telemetry: Upgrades monitoring tools (nvidia-smi and DCGM exporters) to report scoped memory consumption and compute utilization per container fraction rather than misleading whole-GPU aggregates.
• Open-Source Governance: The GPU Fractioning Operator is hosted under the KAI Scheduler GitHub organization and integrates natively via a single Helm configuration flag in KAI Scheduler v0.18.0.

Architectural Comparison: MIG vs. Legacy Time-Slicing vs. CUDA 13.4 NvFractions

The table below compares how Kubernetes platform teams partition data center GPUs across hardware MIG, legacy Device Plugin time-slicing, and the new CUDA 13.4 + KAI Scheduler v0.18.0 NvFractions stack.

Inside the GPU Fractioning Operator and Three-Tier Memory Policies

In previous Kubernetes deployments, running multiple PyTorch containers on a single shared GPU created a hidden failure cascade. Because standard deep learning caching allocators inspect total physical device memory via cuMemGetInfo at startup, each container assumed it owned the entire 80 GB or 141 GB card. When two pods scaled their KV caches simultaneously, the second allocation collided with the first, crashing production inference workers.

With CUDA 13.4 and the GPU Fractioning Operator, the driver virtualizes both the memory reporting and the enforcement boundary for each Kubernetes container namespace. When a pod queries available device memory, CUDA 13.4 reports only the slice allocated to that container—preventing PyTorch or vLLM from over-allocating its memory pool during initialization.

Through Pod annotations managed by the GPU Fractioning Operator and scheduled by KAI Scheduler v0.18.0 (enabled via the NvFractions Helm flag), platform teams can assign workloads to one of two compute modes (sm-sharing via CUDA MPS for concurrent spatial execution, or time-sharing for temporal multiplexing) alongside three distinct memory governance policies:

1. Exact Memory Ceiling: The pod is granted a fixed VRAM budget (for example, 16Gi). Any cudaMalloc call that exceeds the 16 GB boundary immediately returns an out-of-memory error strictly inside that container’s process space, leaving neighboring pods on the same GPU completely unaffected.
2. Guaranteed Minimum With Burstable Maximum: Designed for elastic batch inference or notebook sessions, the scheduler reserves a guaranteed baseline (such as 12Gi) so the pod can never be starved, while permitting the container to burst up to a higher ceiling (such as 32Gi) whenever contiguous unallocated VRAM is available on the physical card.
3. Portion-Based Fractional Allocation: Administrators specify a clean percentage or fractional ratio of the card (such as 0.25 of a GPU), automatically scaling both compute share and memory ceiling across heterogeneous node pools (e.g., L40S, H100, and B200 nodes) without hardcoding byte counts.

Beyond memory enforcement, CUDA 13.4 fixes observability blindness. In legacy shared-GPU setups, running nvidia-smi inside a container displayed the aggregate memory usage of all processes on the physical card, breaking horizontal pod autoscalers (HPA) that relied on per-pod VRAM metrics. The new stack scopes telemetry directly to the assigned fraction, giving FinOps and SRE teams accurate per-pod utilization metrics.

Migrating Multi-Tenant AI Clusters to Memory-Enforced Fractions

For cloud infrastructure and MLOps teams looking to consolidate underutilized GPU nodes without risking noisy-neighbor OOM outages, adopting CUDA 13.4 and KAI Scheduler v0.18.0 provides an immediate path to higher cluster density.

The 4-Step Enterprise Implementation Playbook

1. Upgrade Node Drivers to CUDA 13.4 and Enable NvFractions in KAI v0.18.0: Roll out the CUDA 13.4 driver package across non-MIG GPU node pools first, deploy the open-source GPU Fractioning Operator from the KAI Scheduler GitHub repository, and set the NvFractions Helm flag in KAI Scheduler v0.18.0 to activate fractional bin-packing.
2. Pair sm-sharing With Exact Ceilings for Low-Latency Microservices: For production RAG embedding models, rerankers, and small language models (1B–8B parameters) that underutilize tensor cores individually, configure Pod annotations for sm-sharing (MPS) with an Exact memory policy sized 15% above peak KV-cache requirements. This enables true parallel spatial execution across pods with deterministic memory isolation.
3. Assign Burstable time-sharing Policies to Interactive Data Science Pods: For JupyterHub, VS Code dev containers, and exploratory fine-tuning jobs where GPUs sit idle 80% of the time while engineers write code, apply the time-sharing mode with a Guaranteed Min + Burstable Max policy (e.g., 8 GB guaranteed, 40 GB burstable). Developers get interactive access to large VRAM pools during active runs while allowing high node over-subscription.
4. Re-Calibrate vLLM --gpu-memory-utilization Flags: Because CUDA 13.4 exposes the fraction’s memory limit directly to the container runtime rather than the whole card’s capacity, remove brittle hardcoded fractional flags (like --gpu-memory-utilization 0.22) from vLLM or TGI startup scripts and restore standard 0.90 utilization relative to the enforced container slice.