Cost & FinOps for ML/GPUs

We just finished assembling a serious platform — cluster, pipelines, a feature store with its online Redis and materialize job — and ended by naming the thing we’d treated as free all along: every piece of it runs on rented hardware billed by the second, much of it on the most expensive silicon you can rent. We asked what it actually costs and where the money leaks. This lesson answers, and the short version is that the leak is almost never where you’d guess.

GPU spend went from a footnote to one of the fastest-growing lines on enterprise cloud bills, and in 2026 it’s an MLOps responsibility. The FinOps Foundation now runs a dedicated “FinOps for AI” track for exactly this reason. The core skill isn’t haggling on price — it’s understanding where the money actually goes.

The meter runs whether you use it or not

A GPU bills you for every hour it’s allocated, not every hour it’s used. A fleet sitting at 40% utilization is burning more than half its cost on idle silicon. This is the single biggest leak in ML infrastructure. Size a fleet and watch the waste:

That idle slice is where FinOps starts. Before negotiating cheaper GPUs, the question is always: why aren’t the ones we have busy?

The cost levers

• Raise utilization — the highest-leverage lever. Batch inference requests, share GPUs across jobs (time-slicing / MIG partitioning), use a scheduler (Run:ai, Kueue) to pack work, and autoscale to zero so idle endpoints cost nothing.
• Spot / preemptible instances — 60–90% cheaper for interruption-tolerant work (most training with checkpointing, batch jobs). Not for latency-critical serving.
• Reserved / committed-use — commit to a 1–3 year baseline for ~40% off on-demand, for capacity you know you’ll always need.
• Right-size the accelerator — don’t run a small model on an H100. Match the GPU (and its memory) to the job; a fractional or smaller GPU is often plenty.
• Compress the model — quantization and distillation cut both the GPU needed and the per-inference cost.

Track two numbers

You can’t optimize what you don’t measure. The two metrics that turn cost into a decision:

• Cost per training run — so “should we retrain nightly?” has a dollar answer (ties directly to retraining cadence).
• Cost per 1,000 inferences — the unit economics of serving. If it exceeds the revenue per 1,000 predictions, the model loses money at scale, no matter how accurate.

In one breath

GPU cost is now MLOps’s job, and the skill isn’t haggling on price — it’s knowing the meter bills for every allocated hour, not every used one, so the biggest leak is low utilization (a fleet at 40% wastes over half its spend on idle silicon); the levers, in order of leverage, are raise utilization (batch, GPU-share, autoscale to zero), then spot/preemptible instances for interruption-tolerant training (60–90% off), reserved capacity for known baselines (~40% off), right-sizing the accelerator, and compressing the model — all of it made visible by tracking two numbers, cost per training run and cost per 1,000 inferences, the latter being the serving unit economic that can sink a model more accurate than it is profitable.

Practice

Before the quiz, reason about the counterintuitive core. A teammate proposes cutting the bill by negotiating a cheaper per-hour GPU rate. Using the allocated-not-used idea, explain why raising utilization on the GPUs you already have almost always beats a price discount — and which lever takes idle endpoint cost all the way to zero. Then the unit-economics gate: a model is 4% more accurate but costs $0.012 per 1,000 inferences against $0.009 of revenue — should it ship, and why is “but it’s more accurate” not the deciding factor?

A question to carry forward

Notice what kind of accountability this lesson was about: a dollar one. We learned to make the platform cheap — to tag spend, watch utilization, gate a model on cost-per-inference. And cost is real accountability, the kind a finance team enforces with a budget alert.

But it is not the only bill a production model runs up, and the others don’t show up on a cloud invoice. The same model that’s cheap, fast, and accurate can quietly deny loans to one group at twice the rate of another, can’t explain a single decision when a regulator asks, and may already be out of compliance with a law that took effect while you were optimizing GPU hours. Those costs are paid by people, not by your AWS account — and increasingly, by your legal team. So the question to carry forward is the accountability that doesn’t fit on a dashboard of dollars: once a model is performant and affordable, how do you make it fair, explainable, and compliant — and operate those properties the way we operate everything else? That is responsible-AI ops, and it is the next lesson.