Product3 min read

ByCostGraph Team

GPU costs by model and workload

Compare GPU costs with utilisation and trace spend to the workloads using each card.

Ask about this postClaudeChatGPTPerplexity

Running inference or training workloads means keeping track of both GPU spend and how much of that capacity your workloads use. That gets harder when you use several GPU models, share cards between workloads, or mix spot and on-demand nodes.

We've improved GPU cost tracking in CostGraph to bring those details together. You can compare each model's estimated hourly rate with its utilisation, trace costs to Kubernetes workloads, review AutoPilot's options for a cheaper GPU, and approve GPU partition changes for AutoPilot to apply.

What affects GPU costs

GPU models differ in price, memory, and capabilities. A workload that fits on one card may need more memory or a different feature on another.

Sharing also changes the calculation. A Multi-Instance GPU (MIG) partition is a hardware-isolated slice of a card. Time-slicing lets several workloads share one physical GPU. Cost allocation needs to account for the share each workload requests.

Utilisation helps you assess whether that capacity fits the workload. Compare it with cost to find idle cards or workloads that may fit a smaller GPU.

GPU pricing and utilisation

CostGraph uses a provider's separate GPU rate where one is available. For bundled machines, we estimate the GPU's share of the instance price. GPU hours and the spot and on-demand mix contribute to the cost estimate.

GPU spend follows the cards you actually run. Month-to-date spend stops at the last time CostGraph saw each GPU, and the projected month counts only GPUs seen this month, so a card you removed no longer adds to your GPU spend.

In Kubernetes, costs follow the GPUs a container requests, including its share of a card when that share is known. You can inspect these costs by cluster, namespace, and workload. A cluster's node pools tab lists each pool with its spot and on-demand mix and its node count, and each pool's page shows its CPU, memory, and GPU rates.

See your GPU fleet

Connect your cluster or install the CostGraph agent on your GPU hosts. Enable GPU metric collection, then open GPUs to see the cards reporting to CostGraph.

The page brings each card's model, location, estimated cost, GPU utilisation, and memory use together. Our GPU visibility post walks through the inventory and usage views.

GPU recommendations with AutoPilot

For eligible workloads, AutoPilot looks for a GPU with a lower hourly rate, enough memory for the measured demand, and compatible capabilities. GPU model changes require GPU sizing and placement to be enabled on the workload, along with a compatible Karpenter NodePool.

Shared cards, partitioned GPUs, and workloads that must move together need additional checks. AutoPilot explains when those constraints prevent a change.

The AutoPilot GPU guide covers setup and requirements.

Share GPUs with approved partition changes

CostGraph also recommends how to share GPUs. A sharing recommendation is a plan: the target MIG partition layout for each node, the steps to get there, and an estimate of what draining those nodes costs. CostGraph only offers a plan where your cloud and GPU model can make the change. A shared layout on spot nodes stays a recommendation. Clusters that schedule GPUs through Kubernetes Dynamic Resource Allocation are covered too, and GPU counts reflect the physical cards on each node.

AutoPilot applies a plan only after someone on your team approves it, and only on the nodes they confirm. Each node's approval covers that exact change. With GPU changes turned on in the CostGraph operator, AutoPilot works through the approved nodes one at a time: it cordons and drains the node, applies the new layout, checks that the GPUs report the new partitions, and puts the node back into service. The operator fetches approved work from CostGraph, so your cluster doesn't need to accept inbound connections.

AutoPilot changes partitions on the node itself, on EKS, OKE, and DigitalOcean Kubernetes. On GKE and AKS, which replace a node pool to change its layout, plans stay recommendations. The AutoPilot GPU guide covers setup.

See your own GPU costs

Compare GPU costs with utilisation to find idle or underused capacity. Book a demo and we'll walk through your workloads with you.

Frequently asked questions

Put your bill to work

Connect a provider, watch your spend attribute itself, ask GraphAI where the money’s going.