For some AI deployments, deliverable electricity is becoming a binding co-constraint

At 09:00 on launch day, the model is ready and the racks are not.

Imagine a team with enough budget for a 4,000-accelerator inference cluster. At a planning draw of 700 watts per accelerator, the accelerators alone need 2.8 megawatts. Add:

• CPUs,
• memory,
• networking,
• storage.

Suppose the complete IT load reaches 3 megawatts. With a power usage effectiveness, or PUE, of 1.25, the facility needs 3.75 megawatts at the utility connection.

That load, held continuously for a year, consumes about 32.9 gigawatt-hours. At an illustrative electricity price of 10 cents per kilowatt-hour, the energy alone costs roughly $3.3 million.

The hardware may arrive in six months. The substation, transmission work and permission to draw that load may arrive several years later.

You can buy more chips. You cannot install a transformer by increasing the purchase order.

That is the wall. Electricity.

The grid became the bottleneck

Power is rate; energy is accumulation

Power and energy are related, but they are not the same thing. Power is the rate at which equipment draws electricity, measured in watts or megawatts. Energy is the accumulated amount, measured in watt-hours or terawatt-hours.

A one-megawatt load running for one hour consumes one megawatt-hour. Running it for a full year consumes 8.76 gigawatt-hours.

A data center needs both. It needs enough power capacity to survive its busiest moment, and enough energy delivered over time to serve customers.

A site can have plenty of annual renewable energy on paper and still lack the firm, local capacity to run a new cluster at 3 a.m. on a windless night.

PUE makes the boundary slightly less obvious. A PUE of 1.25 means the facility uses 1.25 units of electricity for every unit consumed by computing equipment.

The extra 0.25 pays for:

• cooling,
• pumps,
• fans,
• power conversion,
• other overhead.

It does not mean the cooling system consumes 125 percent on top of the IT load.

Deliverability is a physical chain

The physical chain is unforgiving. A developer needs:

• an interconnection agreement,
• a substation with capacity,
• transmission that can carry the load,
• equipment to step voltage down,
• a site that can dissipate the heat.

A GPU shipment solves only one part of that chain.

In many attractive data-center markets, interconnection queues and large electrical-equipment lead times are measured in years.

Moving to a power-rich region can help, but then these enter the bill:

• fiber,
• latency,
• water,
• labor,
• permitting,
• local politics.

Scale changes the planning problem

The scale is already large enough to matter outside the technology industry. The International Energy Agency’s Energy and AI analysis estimated global data centers consumed about 415 TWh in 2024 and projected roughly 945 TWh by 2030 in its base case.

Those figures cover all data centers, not AI alone, and they are projections rather than meter readings from the future. The important point is the slope: data-center demand is becoming a large, concentrated new load.

A viral comparison saying that one AI query uses a thousand times the electricity of a web search is not a useful planning constant. The answer depends on:

• the model,
• prompt length,
• generated output,
• hardware,
• batching,
• caching,
• utilization,
• what counts as the competing search operation.

The number may be directionally useful as a warning. It is not a capacity plan.

Inference turns a one-time cost into a permanent load

Training gets the headlines because it is spectacular and easy to narrate: a giant cluster runs for weeks, the model is released, and everyone talks about the bill.

Inference means running those trained weights to produce an answer. For a particular model release, training is a lumpy, one-time event. Inference is a meter that keeps running for every customer, every day.

The multiplication is what changes the economics.

Every request adds work

Take a hypothetical serving tier drawing 1.3 megawatts at the facility boundary while handling 200 requests per second. If the system is steady, its energy per request is 1,300,000 joules per second / 200 requests per second, or 6,500 joules per request.

That is 6.5 kilojoules. If the response contains 100 tokens, where a token is a small piece of text processed by the model, the average is 65 joules per output token.

One million such requests consume 6.5 gigajoules, or about 1.8 megawatt-hours. At 10 cents per kilowatt-hour, that is approximately $180 of electricity.

The electricity is only one part of the serving cost. The following may cost more:

• hardware depreciation,
• staff,
• networking,
• software.

The physical load is real, and it scales with traffic.

The calculation also shows why averages can lie. A service handling 200 requests per second at lunchtime and almost nothing overnight has a different capacity problem from one handling 200 requests per second continuously.

Interactive systems also need spare capacity for bursts. A cluster sized to its daily average will turn a traffic spike into a queue.

Training eventually loses its lead as usage accumulates. Suppose, purely as an accounting example, that a training run consumes 5 GWh and one served query consumes 50 kJ.

Serving 360 million queries consumes the same 5 GWh, because 360 million multiplied by 50 kJ is 18 trillion joules, or 5 GWh. A heavily used product can reach that volume without the model feeling like a global phenomenon.

The often-repeated claim that inference represents 80 to 90 percent of AI compute should be treated as a rough estimate for particular fleets and accounting boundaries, not a law of nature.

A research lab retraining models every week may be training-heavy. A consumer assistant with millions of daily users may be serving-heavy.

The durable claim is simpler: once a model has substantial traffic, repeated inference can outweigh its one-time training cost.

Longer reasoning means more serving work

Reasoning models make the multiplication more aggressive. Asking a model to spend more tokens considering a problem means more forward passes before the answer arrives.

Long prompts also increase the work needed to process the input. A KV cache, the stored key and value activations for tokens already processed, avoids recomputing some context during generation, but it consumes memory and memory bandwidth.

More context is not free merely because the model accepts it.

More tokens do not translate into exactly proportionate energy in every system. Prefill, decoding, memory movement, communication between accelerators and idle capacity all contribute.

The practical rule is still sound: a response that requires five times as many generated tokens generally asks the serving system to do substantially more work. “Thinking longer” is a product decision with a power consequence.

The useful target is joules per good answer

Measure the whole serving path

The metric worth putting beside latency, quality and cost is energy per successful answer. The inference metrics should include:

• request volume,
• input and output tokens,
• throughput,
• queue time,
• accelerator utilization,
• power where possible.

Model size is a poor substitute. Two models with the same parameter count can move different amounts of data and use different hardware paths.

Optimize the serving path

Routing is often the cleanest win. Suppose 80 percent of requests can be answered by a smaller model using 30 kJ, while 20 percent need a larger model using 150 kJ. The weighted average is 54 kJ per request.

Sending every request to the larger model would use 150 kJ, so the mixed design cuts average energy by 64 percent, assuming the router does not create retries or unacceptable quality failures.

Model routing is not just a latency technique; it is a way to reserve scarce power for the questions that need it.

Quantization can reduce the number of bytes moved between memory and compute by representing weights with fewer bits. That can lower memory pressure and energy, especially when the hardware has a good low-precision path.

It is not magic compression. For a particular model, the benefit can be erased by:

• dequantization work,
• unsupported operations,
• quality loss.

Measure the complete serving path rather than celebrating a smaller checkpoint.

Batching improves the picture in another way. Running several requests together can keep expensive hardware busy and amortize fixed overhead.

But waiting to form a batch increases latency, especially for a user staring at a spinner. Continuous batching helps a serving system admit and retire requests dynamically, yet it still requires a choice between throughput and tail latency.

A 20 percent energy improvement that turns the 99th-percentile response from two seconds into twelve is not a free win.

Caching repeated prefixes and answers can avoid recomputing work. It also has several consequences:

• it consumes memory,
• it needs invalidation rules,
• it can create privacy problems if one user’s context is accidentally reused for another.

Output limits and reasoning budgets are similarly effective but visible to users: fewer tokens may reduce energy while also reducing answer quality.

FLOPs are not joules. Arithmetic is only one part of the bill. Energy also goes into:

• moving data through memory,
• sending activations between accelerators,
• running an underfilled device.

A model with fewer nominal operations can lose to a larger model that runs efficiently on the available hardware. The only honest comparison is the measured service at its target quality, traffic shape and latency.

Efficiency can increase total demand

If a service handles 10 million requests at 100 kJ each, it consumes about 278 MWh. Improve the system to 40 kJ per request, then let traffic grow to 30 million requests, and total energy rises to about 333 MWh.

The product became 2.5 times more efficient per answer, using 60 percent less energy, and still used more electricity. Cheap intelligence invites more intelligence.

The strongest objection is partly right

Power is not a permanent natural limit. Accelerators are becoming more capable per watt.

Efficiency can also improve through:

• better cooling,
• higher utilization,
• model compression,
• improved serving software.

These can reduce energy per token.

New supply can come from:

• new generation,
• transmission,
• batteries,
• on-site power.

If those improvements arrive faster than demand grows, the bottleneck moves.

That does not make electricity irrelevant. The constraint is marginal, local and temporal.

A company does not need “enough energy somewhere on Earth.” It needs deliverable power at a permitted site, on the date its cluster is ready, with reliability acceptable to its service.

A power-purchase agreement can change emissions accounting, but it does not by itself make a substation deliver more firm capacity at a chosen hour.

Other constraints remain, including:

• chips,
• capital,
• networking,
• people.

The argument is not that electricity has replaced every other bottleneck. It is that power is now a binding co-constraint, and one that software teams often discover after they have finished the model and bought the hardware.

What to do on Monday morning

• Measure at the wall. Record facility or rack power alongside:

  • requests per second,
  • input tokens,
  • output tokens,
  • batch size,
  • queue time,
  • model version.

  Accelerator thermal design power is a ceiling or planning figure, not your energy-per-answer measurement. Compare a warm steady run with an idle period and a burst. Include the PUE boundary so that two teams are not comparing IT-only power with whole-facility power.

• Build a power forecast, not just a GPU forecast. For a steady serving tier, facility power = requests per second × joules per request. If energy is measured per token, use tokens per second × joules per token.

  Run the calculation for:

  • average traffic,
  • peak traffic,
  • a growth case.

  Include:

  • retries,
  • replicas,
  • failover capacity,
  • long-context requests,
  • the reasoning budget.

  A cluster that needs 3.75 MW at peak should not be described as a 3 MW deployment because the spreadsheet preferred a round number.

• Make energy a release metric. Put the following beside one another:

  • joules per successful answer,
  • p95 latency,
  • task success,
  • cost.

  Test each separately:

  • a smaller routed model,
  • quantization,
  • caching,
  • a lower output budget.

  Keep the change only when quality remains inside its agreed boundary. This prevents the familiar failure where a team saves power by quietly making the product worse.

• Treat power delivery as an external dependency. Ask the data-center provider or utility for the date and amount of power actually deliverable, not the site’s ultimate nameplate capacity.

  Confirm:

  • the ramp schedule,
  • redundancy,
  • curtailment rules,
  • who pays for the electrical work.

  If the proposal depends on on-site generation, include:

  • fuel,
  • permitting,
  • emissions,
  • maintenance,
  • backup behavior.

  “We can add another region later” is not a plan if every attractive region has the same queue.

The first symptom of a power-constrained AI service is usually not a blackout. It is rising queue age, worsening p95 latency and an autoscaler requesting replicas that never become available.

A power cap may throttle clocks, so GPU utilization looks healthy while tokens per second falls. If the dashboard tracks only utilization and request cost, the incident can masquerade as a software regression.

The next AI advantage will still come from better models. But the winning system will answer a harder operational question: how much useful reasoning can it deliver per joule, at the hour and place where users actually ask for it?

Intelligence matters. The grid decides how often you get to use it.