how do MIGs on Kubernetes work?

Multi-Instance GPU, or MIG, is a way to slice up a single GPU to make better use of it. It isn't new by any stretch. NVIDIA introduced it with the Ampere-based A100 in May 2020.

Enabling it in Kubernetes is a bit of a different story. In this article, we'll walk through how MIG works in Kubernetes, how to enable it, and how it differs from GPU time-slicing, which I covered in an earlier article.

Splitting a single GPU

Starting with the A100, supported NVIDIA GPUs can expose smaller, independent partitions with dedicated compute and memory. Support depends on the GPU model; being an Ampere GPU alone doesn't make a card MIG-capable.

For example, an A100 80 GB can be configured as two 3g.40gb devices. Each has three compute slices and 40 GB of nominal memory. We'll come back to what a slice contains shortly. For now, the useful detail is that these devices already exist before we submit a Pod.

From here, a Pod can request one of those devices through its Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
    name: inference-a
spec:
    replicas: 1
    selector:
        matchLabels:
            app: inference-a
    template:
        metadata:
            labels:
                app: inference-a
        spec:
            containers:
                - name: inference
                  image: your-registry/inference:tag
                  resources:
                      limits:
                          nvidia.com/mig-3g.40gb: 1

Replace the image with your inference service. The 1 requests one complete device of this profile. This resource name requires the device plugin's mixed strategy, which exposes each profile through its own resource name. Submitting the Pod doesn't create the partition.

What happens when we allocate it?

Click Allocate Pod A below. Its container receives access to one device's compute and memory. Allocate Pod B to give it the second device, then release Pod A.

Each Pod gets isolated resources. Pod A can use all of its own compute and memory, but it cannot take Pod B's reserved allocation. An idle partition doesn't lend its resources to a busy neighbour.

NVIDIA implements this through dedicated paths through the memory system. Each GPU instance has its own L2 cache banks, memory controllers and DRAM address buses, as well as dedicated compute. That provides memory capacity and bandwidth isolation. If Pod A fills its memory or saturates its memory bandwidth, it doesn't consume Pod B's allocation.

Releasing Pod A makes its device available again. The partition remains configured, and Pod B keeps running in its own partition.

How does this differ from time-slicing?

MIG is often mentioned in the same breath as time-slicing as a way to maximise GPU usage. Let's keep the same two workloads, but change how they share the GPU.

With time-slicing, NVIDIA's device plugin can advertise multiple replicas of a physical GPU. Each replica grants shared access to that GPU. Work from the workloads is interleaved on shared execution resources; a replica does not reserve its own portion of GPU memory.

Notice that the memory allocations on the left stay visible when the compute changes colour. Time-slicing doesn't mean unloading one application's memory every time another gets to run.

NVIDIA's time-slicing implementation provides shared access without MIG's memory and fault isolation between replicas. Requesting additional replicas also doesn't guarantee proportionally more compute.

What is inside that partition?

A GPU instance, or GI, contains compute slices and dedicated memory resources. A compute instance, or CI, selects compute slices within that GI to execute work.

Click View partition to pull our 3g.40gb instance apart. The CI contains all three compute slices in this example. The memory, L2 cache and memory controllers belong to the enclosing GI.

We can divide a GI's compute into several CIs, but those CIs share their parent GI's memory and engines. Creating another CI does not create another independent memory partition. Our example uses one CI per GI.

Going back to the A100 example, MIG has seven compute slices and eight memory slices available. Each compute slice contains multiple streaming multiprocessors, or SMs.

Our 3g.40gb profile combines three compute slices with four memory slices. Two such instances fully allocate all eight memory slices, leaving one compute slice unassigned.

Click Slice to create the first 3g.40gb partition, then click it again to create the second.

These are resource allocations on an idle GPU. MIG uses predefined profiles, so we can't cut it at arbitrary positions or ask for any percentage we like. Other GPU models support different profiles. The memory in a profile name is nominal capacity; the amount an application can allocate is slightly smaller.

How Kubernetes connects the pieces

Allocation in Kubernetes is often a multi-step process involving a few components.

In NVIDIA's Kubernetes integration, GPU Feature Discovery supplies labels describing the hardware. Device assignment follows the Kubernetes device plugin workflow described above.

The Deployment earlier requests nvidia.com/mig-3g.40gb. That resource name requires the device plugin's mixed strategy, which we'll configure in the next section. You may also encounter single, which exposes identically configured MIG devices on a node as nvidia.com/gpu.

We use the device plugin workflow in this example, but Dynamic Resource Allocation might be worth looking into.

Configuring our two devices

Let's configure the two 3g.40gb devices we've been using. We'll assume the node has one A100 80 GB, a supported driver and GPU Operator installed with MIG Manager enabled. The commands below are an example configuration, not output from a hardware test.

The primary way to control how MIG devices are made available to Kubernetes is through the device plugin's MIG strategy. The layout determines which partitions MIG Manager creates on the GPU.

Before changing the layout, arrange a maintenance window for the GPU workloads on that node. MIG Manager stops GPU workloads during reconfiguration, and enabling MIG may require a GPU reset or a node reboot.

Expose the profiles

GPU Operator stores its configuration in a resource called a ClusterPolicy. Find the name of yours first.

kubectl get clusterpolicies.nvidia.com

In this example, it is named cluster-policy. We set its MIG strategy to mixed so the device plugin advertises profile names such as nvidia.com/mig-3g.40gb. Replace cluster-policy in the command if your policy has a different name.

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type=merge \
  -p '{"spec":{"mig":{"strategy":"mixed"}}}'

This changes the strategy for the Operator's managed nodes. We still need to select a partition layout for our A100 node.

Choose the layout

The MIG Manager reads the node label nvidia.com/mig.config to find out which layout to apply. Its value names a configuration in MIG Manager, rather than an individual device.

Here we use all-3g.40gb, which configures our A100 80 GB as two 3g.40gb devices. That named configuration must be available in your installation. GPU Operator versions differ in how they provision these configurations, so check the MIG Manager configuration documentation for your version.

Set GPU_NODE to the name of your A100 node, then apply the label.

GPU_NODE=your-a100-node

kubectl label node "$GPU_NODE" \
  nvidia.com/mig.config=all-3g.40gb --overwrite

MIG Manager uses NVIDIA's MIG partition editor to apply that layout. Updating the label requests the change; it doesn't mean the devices are ready yet.

Wait for the devices

We need to check two things before submitting a workload. MIG Manager reports its progress through the nvidia.com/mig.config.state label. The devices also need to appear in the node's allocatable resources after the device plugin discovers them.

This command uses jq to pull out both values.

kubectl get node "$GPU_NODE" -o json | jq '{
  state: .metadata.labels["nvidia.com/mig.config.state"],
  resources: (.status.allocatable | with_entries(
    select(.key | startswith("nvidia.com/"))
  ))
}'

For our single GPU with two devices and no additional time-slicing, the expected result is below. Wait until the state is success and the resource count is 2.

{
    "state": "success",
    "resources": {
        "nvidia.com/mig-3g.40gb": "2"
    }
}

The allocatable count is the node's usable capacity, not a count of free devices. It remains 2 when Pod A starts. The scheduler tracks the Pod's request separately when deciding whether another Pod will fit.

We can now submit the Deployment from the beginning of the article. Its limit of nvidia.com/mig-3g.40gb: 1 asks for one device. Kubernetes uses that limit as the request when no request is specified. If we supply both, their values must match.

When to use MIG

In general, use MIG when several workloads can each fit inside a GPU partition and need dedicated compute and memory. Inference services that don't need a whole GPU are a good example.

For an LLM, leave room for the model weights, KV cache, temporary buffers and serving engine allocations.

CERN's paper on GPU sharing with Kubernetes is a good read, as it distinguishes between pooling GPUs across users and sharing a single GPU.

Two MIG devices still share the same physical GPU and host. If that GPU or host fails, both workloads can be affected.

Where's the catch

We know most marketing sucks, but if you're curious how CostGraph can help you optimise and attribute your GPU and Kubernetes spend, check us out and get started for free :)