Your GPUs are mostly idle: FinOps for the AI era

At 2:17 a.m., the endpoint is receiving no traffic. The GPU has loaded the model, the Kubernetes node is healthy, and the cloud bill is advancing by the minute.

At 9:00 a.m., traffic returns. The same GPU now spends much of its time waiting for CPU preprocessing, disk reads, or the next request in a badly shaped batch. The dashboard says the service is “up.” The invoice says nothing about that distinction.

This is the uncomfortable part of AI infrastructure: the scarce thing is not always the GPU-hour. It is useful work extracted from the GPU-hour.

Most AI cost programs start with the rate card. They compare cloud providers, negotiate discounts, and debate spot versus reserved capacity. Those choices matter. But if an accelerator is allocated for 720 hours and does useful work for 288 of them, a 20% price discount does not cure the main problem.

You are buying time. Start by using the time.

The bill follows allocation, not effort

FinOps means financial operations for cloud: connecting infrastructure spending to the teams, workloads, and business outcomes creating it. For a normal CPU service, waste is often visible as an oversized instance. GPUs make the same mistake much more expensive because one card can cost as much per hour as an entire small server.

For provisioned GPU instances, providers generally bill the instance or attached accelerator while it is running, regardless of kernel activity. Managed inference APIs may instead charge per request, token, image, or other unit. Reserved or committed capacity—an agreement to pay for capacity over a term—can also create charges independently of current allocation.

For a provisioned instance, “allocated” means provisioned and available to your workload. “Utilized” means doing the work you care about, such as processing a batch or generating tokens. A GPU does not become free because its kernels are waiting.

Consider an intentionally simple example:

• Eight GPUs are allocated continuously.
• The effective provisioned rate is $2.50 per GPU-hour.
• The month has 30 days.

The bill is 8 × 24 × 30 × $2.50, or $14,400.

Now suppose the GPUs are doing useful work only 40% of the time. That is 2,304 useful GPU-hours, so the effective cost is $6.25 per useful GPU-hour. If better batching and scheduling raise useful utilization to 70%, the same eight GPUs produce 4,032 useful GPU-hours. The effective cost falls to about $3.57 per useful GPU-hour, even though the hourly price has not changed.

If the workload can actually be served with five GPUs at 70% utilization, the monthly accelerator bill becomes $9,000. That saves $5,400 without negotiating a cent.

The word “actually” is doing important work there. Five GPUs may fail the latency target during the afternoon peak, may not fit the model in memory, or may leave no room for a node failure. Utilization is a lever, not a magic number.

The broader spending context needs labels. Gartner’s November 2025 forecast projected worldwide AI spending of $1.76 trillion in 2025 and $2.52 trillion in 2026. Those are forecasts, not observed invoices. McKinsey’s 2025 analysis estimated about $5.2 trillion of AI-related data-center capital expenditure over the 2025–2030 build-out in its base case, within roughly $7 trillion of total global data-center capital expenditure. The $5.2 trillion figure is cumulative AI-attributable data-center capex, not annual AI spending or total AI spending. Forecasts will move. The arithmetic of an idle accelerator will not.

“GPU utilization” is not one thing

A beginner sees a dashboard reading 12% and assumes the GPU is wasting 88% of its time. An expert asks what the metric measured, over what window, and whether the remaining time was needed for a latency objective.

Tools such as nvidia-smi expose a useful device-busy signal: roughly, how much of a sampling interval had one or more GPU kernels running. That is not the same as useful application throughput. A GPU can show high activity while repeatedly doing small, inefficient kernels. It can also show low activity because the CPU, input pipeline, network, or scheduler is late.

Track at least four separate facts:

1. Allocated time: how long the instance or GPU was provisioned.
2. Device busy time: how often the device ran kernels.
3. Work completed: requests, tokens, images, examples, or records successfully processed.
4. Service quality: queue depth, throughput, and p50, p95, and p99 latency.

The fourth matters because a low utilization number may be healthy headroom. An online fraud model that must answer in 40 milliseconds cannot fill every last scheduling slot if doing so creates a queue during a burst.

The useful question is not “How do we reach 100%?” It is “How much accepted work do we get for each dollar while meeting the service objective?”

For inference, record fully allocated cost per 1,000 successful inferences: (GPU + CPU + storage + network + orchestration + allocated commitment/shared costs) ÷ successful inferences × 1,000. Count all capacity held during quiet periods, retries, and failed requests in the numerator. Use successful outputs—not attempts—as the denominator. For a managed request-priced API, use its actual request or token charges and allocate any surrounding infrastructure.

For training, record the fully allocated cost of a run, not only the GPU line item. A run that uses eight GPUs for ten hours costs 80 GPU-hours before CPU workers, storage, network transfer, and orchestration are counted. If the run fails after nine hours because a data loader crashed, it consumed 72 GPU-hours and produced zero usable output; include those 72 hours and ancillary costs in the failed run’s cost.

Where the idle time comes from

Idle capacity usually has a cause you can name. That makes it fixable.

Demand idle

An endpoint with highly uneven traffic may need eight GPUs at noon and none at midnight. Keeping all eight warm avoids cold-start delay, but it also charges you for a quiet night.

Autoscaling can reduce this waste, sometimes to zero replicas, but only when the application can tolerate startup time and the platform truly releases the underlying capacity. Scale on a useful signal such as queue depth or in-flight requests, with a latency guardrail. GPU utilization alone is a poor trigger because it reacts after the queue has formed.

Use separate policies for baseline and bursts. Keep the minimum capacity needed for the normal load. Add capacity when the queue or forecast says it is needed. Scale down slowly enough to avoid removing a replica, discovering a burst, and adding it back every few minutes.

Pipeline idle

Training and inference pipelines often starve the accelerator. The GPU waits while CPUs decode images, tokenize text, fetch objects from remote storage, or prepare the next batch.

The first symptom is often a sawtooth trace: short bursts of high GPU activity separated by regular gaps. Increasing the GPU count makes the graph more impressive and the bill worse.

Profile the handoff between stages. Measure data-loader wait time, CPU saturation, storage throughput, network transfer, kernel launch overhead, and synchronization barriers. Faster storage, more appropriate worker counts, prefetching, local caching, or a better input format can produce more work from the same cards. Adding GPUs before fixing the slowest stage simply creates a larger waiting room.

Anyscale’s reported GPU-utilization case measured a containerized pipeline at 12.5% device utilization: insufficient CPU capacity starved the pipeline and left 224 GPUs idle. The lesson is not that every pipeline has that shape. It is that “we have enough GPUs” is not evidence that the pipeline can feed them.

Scheduling and fragmentation idle

A cluster can have spare capacity and still be unable to place a job. One model needs a full GPU. Another needs a particular memory size. A third has reserved four cards on a node but uses only two. The scheduler sees reservations and constraints; the invoice sees running machines.

Packing compatible workloads onto the same node reduces this stranded capacity. Time-slicing lets workloads take turns on a device. NVIDIA MIG, where supported, divides a GPU into hardware-backed instances with dedicated portions of resources. These approaches are not interchangeable.

MIG gives stronger partitioning and more predictable interference, but it can fragment a large card. A small slice may be free while a job needing a larger slice waits. Time-slicing can improve average sharing, but workloads compete for execution time and may affect one another. Test isolation, memory behavior, and tail latency rather than choosing the fashionable option.

Model and hardware mismatch

A small classifier does not automatically deserve the same accelerator as a large generative model. Conversely, choosing a cheaper card can make a memory-bound model spill, reduce throughput, or force a slower batch shape.

Right-size with measurements. Compare throughput at the required quality and latency, not just the nominal hardware price. Lower precision, quantization, a smaller model, or distillation may reduce memory and increase throughput, but each can change accuracy or output behavior. The quality evaluation is part of the cost calculation.

Batching is powerful because it changes the economics

A GPU is good at parallel work. Sending one tiny request at a time pays fixed overhead repeatedly: kernel launches, scheduling, memory movement, and sometimes loading the same weights through the execution path. Batching lets one operation handle several examples together.

Suppose an inference service processes 100 requests per second at $3 per GPU-hour. Its GPU cost is approximately $0.0083 per 1,000 requests if it sustains that rate for the full hour. If better batching raises throughput to 250 requests per second on the same GPU, the cost falls to approximately $0.0033 per 1,000 requests. The hourly bill is unchanged. The useful output is not.

That improvement has a price. Waiting to form a batch adds queueing delay. A batch size that is excellent for offline image classification may violate the p99 target for an interactive assistant. Dynamic batching needs a maximum batch size and a maximum wait time. The first bound protects memory; the second protects users.

This is why batch and real-time inference are fundamentally different economic choices. Batch vs real-time inference explains the trade-off in serving terms. The cheapest prediction is often the one you can delay and process in a packed batch. The cheapest interactive prediction is the one that meets its latency target with no unnecessary headroom.

The strongest objection is valid

“Low utilization is not automatically waste,” a platform engineer will say. “I need spare capacity for traffic spikes, failover, and the latency SLO. If I push the number too high, the 3 a.m. page becomes a 9 a.m. outage.”

Correct. Chasing 100% utilization is a bad operating policy.

The answer is to optimize the boundary, not the average. A service may intentionally run at 45% average device busy time because its peak queue and failover requirements demand it. That is a defensible cost. A service sitting at 45% because it runs eight GPUs overnight for two requests is not the same situation.

Look at utilization by hour and by replica, then compare it with queue depth and tail latency. Keep safety headroom where the SLO requires it. Move interruptible batch work to spare capacity. Use a small reliable baseline and an overflow pool when the workload permits. A cost dashboard without latency and reliability context invites the wrong optimization.

Price reductions still help, but they are multiplicative. A 30% discount on the $14,400 example saves $4,320. Removing three unnecessary GPUs saves $5,400 at the assumed rate, before any further discount. If utilization work lets you buy fewer cards without breaking the peak requirement, it changes the size of the bill rather than merely shrinking its unit price.

What to do on Monday morning

Start with one service or training pipeline. Do not begin by redesigning the whole cluster.

Export the last seven days of allocated GPU-hours, device-busy time, request count or training progress, queue depth, p95 or p99 latency, and billing data. For each resource, collect the effective billed rate after discounts, spot charges and interruptions, reservation or commitment charges, CPU and storage rates, network transfer, and orchestration costs. Allocate shared costs using an explicit rule. Record each training attempt’s outcome—completed, failed, or retried—and tag the team, model, environment, and run.

Raw hours are not dollars. A list of GPU-hours cannot account for effective rates, discounts, commitment charges, or shared infrastructure.

Call this fully allocated infrastructure cost: every direct resource charge and agreed share of common charges assigned to the workload, not just the GPU line.

Then calculate two numbers:

• Fully allocated infrastructure cost per completed training run: sum the GPU, CPU, storage, network, orchestration, commitment, and shared costs for every attempt, including failed and retried attempts, then divide by the number of runs that produced an accepted artifact.
• Fully allocated infrastructure cost per 1,000 successful inferences: sum those same applicable costs, including capacity held during quiet periods and failed or retried requests, divide by successful inferences, and multiply by 1,000.

If you cannot assign an infrastructure dollar or hour to a team and model, fix tagging or ownership before making optimization claims. Cost that cannot be allocated becomes everyone’s problem and nobody’s priority. Cost and FinOps for ML/GPUs has the accounting patterns and a calculator for this step.

Next, inspect the trace. A flat low-utilization line during no traffic points to demand idle. A sawtooth line points toward the input pipeline or synchronization. High device utilization with poor throughput points toward inefficient kernels, memory pressure, or excessive small requests. High utilization with a growing queue points toward overpacking or insufficient capacity, not a victory.

Make one reversible change. Set a maximum batch wait, move an offline job into a shared pool, lower the minimum replicas overnight, or fix the slowest data stage. Record the before-and-after cost per unit of work and the service-quality impact. Keep the change only if the unit economics improve without violating the agreed SLO.

Finally, add a budget alert and an idle-capacity alert. A budget alert tells you that money was spent. An idle alert gives someone a chance to prevent the spend.

When not to optimize utilization

Do not force sharing onto a latency-critical service whose workload is already predictable and well-sized. Do not put a stateful training job on interruption-prone capacity merely to improve a chart. Do not spend two weeks building a custom autoscaler to save less than the engineer time and operational risk it introduces.

And do not trust a single utilization percentage. A GPU at 90% busy may be inefficient if it is doing retries. A GPU at 35% may be exactly right if it protects a strict tail-latency target. The decision belongs to cost per successful outcome, not to a dashboard gauge.

The operational pattern is straightforward: measure allocated time, identify why useful work is missing, improve packing or throughput, then right-size the fleet. Pricing comes after right-sizing, and spot and reserved capacity solve different problems.

Reserve or commit only the measured stable baseline—the capacity needed for ordinary demand. Reserved or committed capacity can lower the effective rate for stable workloads, but unused commitments become waste and the term can lock you into a capacity shape or provider. Include any upfront or fixed charges when comparing the effective cost.

Use spot capacity for checkpointable, retryable, interruptible work. Spot instances can be reclaimed, so quantify interruption rate, lost compute, checkpoint time, recovery time, retry capacity, and availability. A lower sticker price is not a saving if recovery and recomputation erase the discount. Compare those expected costs with the cost of reliable capacity.

For the production details, pair the accounting with ML observability and treat capacity changes as a deployment, with a rollback condition. Inference cost is an architectural choice, not an invoice surprise.