datarekha
Infrastructure June 10, 2026

Your GPUs are mostly idle: FinOps for the AI era

AI spending is heading past $2 trillion while production GPU fleets often run under 50% utilization. The biggest lever on an ML bill isn't a cheaper price — it's the idle silicon you're already paying for.

6 min read · by datarekha · mlopsfinopscostgpuinfrastructure

There’s a number that should bother anyone with a GPU bill: the expensive accelerators you’re renting are, more often than not, mostly idle. A GPU charges you for every hour it’s allocated, not every hour it does useful work — so a fleet at 40% utilization is burning more than half its cost on silicon that’s doing nothing. In 2026, with AI infrastructure spend exploding, closing that gap is the single biggest lever in ML cost management.

The scale makes the waste expensive

The macro picture is staggering: Gartner forecasts global AI spending will reach roughly $1.5 trillion in 2025 and top $2 trillion in 2026, driven by hyperscaler investment in AI data centers and GPU hardware, and longer-term, McKinsey estimates AI data-center capital expenditure could reach about $5.2 trillion by 2030 in its base case. No surprise, then, that cost discipline became urgent: the FinOps Foundation’s State of FinOps 2025 survey — covering organizations responsible for more than $69 billion in cloud spend — found 63% of organizations now track AI spend, up from 31% the prior year.

Utilization is where the money actually leaks

Before negotiating a cheaper GPU-hour, ask why the ones you have aren’t busy. Anyscale’s analysis is sobering: production AI workloads often achieve well below 50% sustained GPU utilization even under load, with one containerized pipeline hitting only 12.5% utilization because 224 GPUs sat idle just to provide enough CPUs. That idle slice — not the unit price — is the prize. The levers, in order of impact:

  • Raise utilization — batch inference, share GPUs (time-slicing / MIG), use a scheduler to pack work, and autoscale endpoints to zero so idle ones cost nothing.
  • Spot / reserved pricing — spot for interruption-tolerant training and batch; reserved for the baseline you always run.
  • Right-size the accelerator — don’t run a small model on a top-end card.

Why AI needs its own FinOps practice at all: traditional FinOps targeted predictable compute, storage, and networking, whereas AI introduces highly variable cost drivers like GPU-intensive training, fluctuating inference demand, and token-based pricing — with average monthly AI spend rising from $62,964 in 2024 to a projected $85,521 in 2025. The FinOps Foundation now runs a dedicated “FinOps for AI” working group defining how AI workloads need allocation, anomaly detection, and forecasting to understand GPU spend.

You don’t have to choose between capability and cost. Most of the bill is idle capacity and oversized hardware — fix those first, and the savings dwarf anything you’d win haggling on price.

Skip to content