How does autoscaling work for ML inference services, and what metrics should drive it?

How to think about it

The short answer

Autoscale an ML inference service primarily on a leading signal tied to the service-level objective, usually per-replica queue wait or backlog, with GPU utilization or model-specific throughput as supporting signals. CPU utilization alone is often the wrong signal because the host CPU can be mostly idle while the GPU is saturated and requests are piling up.

Kubernetes Horizontal Pod Autoscaler, or HPA, can consume custom and external metrics. For a latency-sensitive online endpoint, keep warm capacity available. Use scale-to-zero only when measured cold-start time fits the contract or when the work can wait.

Why CPU gets this wrong

An inference request usually passes through four stages: it arrives, waits in a queue, uses the model, and returns a response. A queue is simply work waiting for an available worker.

The important relationship is between arrival rate and service rate. If requests arrive at rate λ, each replica can process at rate μ, and there are N replicas, the service can keep up only while λ stays below Nμ. Once λ exceeds Nμ, the queue grows. Latency then grows even if the CPU graph looks tranquil.

That happens frequently with GPU inference. The application process submits work to CUDA, waits for the GPU, performs a little preprocessing, and waits again. During those waits, the host CPU may be using 10 percent of its capacity. The GPU, meanwhile, may be running kernels continuously. CPU-based HPA sees spare capacity and declines to add a replica. Users see timeouts.

GPU utilization is closer to the bottleneck, but it is not a complete answer either. A GPU at 95 percent utilization with an empty queue may be healthy and cost-efficient. A GPU at 55 percent utilization with a long queue may indicate that tokenization, image decoding, network input, or CPU preprocessing is the real bottleneck. Adding GPUs will not repair a slow tokenizer.

The useful rule is not “always scale on GPU.” It is “scale on the signal that explains the work users are waiting for.”

A concrete example

Consider a receipt-scanning API. Each pod runs one vision model on one GPU. The team load-tests the exact production request mix and finds that one pod can sustain about 100 requests per second while keeping its p95 latency at or below 200 milliseconds.

At a normal 80 requests per second, one pod looks comfortable:

Measurement	Observed value
CPU utilization	12%
GPU utilization	74%
Median latency	22 ms
p95 latency	47 ms
Queued requests	0

At the morning traffic spike, demand reaches 115 requests per second. The same pod now shows 15 percent CPU utilization and 96 percent GPU utilization. Its queue contains 38 requests, and p95 latency rises to 310 milliseconds.

A CPU-based policy with a 60 percent target does nothing. From the CPU controller’s perspective, the pod is barely working. A queue-wait or backlog policy reacts to the problem users are actually experiencing and adds capacity.

After a second pod is ready, the two pods can handle this request mix comfortably. The key word is ready. A pod that has been scheduled but is still downloading weights or initializing its CUDA context is not capacity.

Queue depth must be interpreted carefully. A target of five queued requests per pod is not a universal magic number. If the average queue depth is 12 across two pods and the target is five, the basic HPA recommendation is approximately:

ceil(2 × 12 / 5) = 5 replicas

That calculation can overshoot if the metric is a global queue, if the backlog accumulated during a short burst, or if a new replica takes 45 seconds to become useful. Per-pod metrics, queue wait time, scale-up rate limits, and readiness checks make the result much safer.

Which metrics should drive scaling?

Signal	Good use	Important limitation
Queue wait time	Protecting a latency SLO and reacting to bursts	Requires the service to measure time before execution
Queue depth	Simple burst and backlog control	Raw counts depend on arrival rate and request cost
GPU utilization	Steady GPU-bound workloads	High utilization can be healthy; low utilization can hide a CPU bottleneck
In-flight requests	Concurrency-limited servers	Requests may have radically different compute costs
Requests per second	Stable workloads with similar request sizes	One large request may cost as much as hundreds of small ones
p95 or p99 latency	SLO monitoring and a safety signal	Latency is a lagging signal and can cause oscillation
Tokens and KV-cache usage	LLM serving	Prompt processing and token generation create different bottlenecks

Queue wait is often better than queue depth. Depth is a count; wait is already expressed in time. Under stable conditions, Little’s law relates the average number of items in a system to arrival rate and time in the system: L = λW. If the arrival rate changes sharply, the same queue depth can imply a different wait time. A queue of 20 requests is not equally serious at 200 requests per second and 5 requests per second.

For large language models, “requests per second” is especially crude. Twenty requests with 2,000-token prompts and twenty requests generating 20 tokens do not consume the same resources. Track time to first token for prompt-processing pressure, inter-token latency for generation quality, waiting and running requests, input and output token throughput, and KV-cache occupancy. KV cache is the memory holding attention state for active sequences. When it fills, requests may wait even though average GPU utilization does not look alarming.

Latency should usually be a guardrail rather than the only scaling trigger. By the time p99 latency has breached the SLO, the queue already exists. A leading signal such as queue wait can scale before the breach, while latency confirms whether the chosen target is actually working.

What the Kubernetes pattern looks like

HPA does not read arbitrary Prometheus metrics by itself. A custom-metrics adapter, an external-metrics provider, or an event-driven autoscaler must make the metric available through the Kubernetes metrics APIs.

This is a representative HPA using a per-pod queue-depth metric. inference_queue_depth is an application metric, not a built-in Kubernetes metric. The adapter configuration that exposes it depends on the monitoring stack.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_queue_depth
        target:
          type: AverageValue
          averageValue: "5"
  behavior:
    scaleUp:
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

The target means an average of five queued requests per pod, not five requests for the whole Deployment. HPA periodically compares the observed value with the target and calculates a desired replica count. It then applies the minimum, maximum, and scaling-behavior rules.

In practice, use more than one signal. Queue wait can drive scale-out, while GPU utilization catches sustained compute pressure. HPA evaluates each metric and generally follows the most aggressive replica recommendation. That prevents a healthy CPU graph from masking a dangerous queue.

The production pattern has two autoscaling loops. The pod autoscaler decides how many model replicas are needed. The cluster or node autoscaler decides whether enough GPU machines exist to run them. If HPA wants eight replicas but the cluster has no available GPUs, the new pods remain Pending and the queue stays high. Increasing maxReplicas does not create hardware, quota, or a faster model download.

Readiness also matters. Do not mark a pod ready when the process has merely started. Mark it ready only after weights are loaded, the CUDA context is initialized, and a health check has proved that the model can serve. Otherwise the load balancer sends traffic to a pod that contributes zero useful capacity.

The senior nuance: scale-to-zero and scale-down

Scale-to-zero saves money, but it changes the latency contract. A request arriving at zero replicas needs an activation path. Knative, KEDA, and similar systems can watch traffic or an external queue and create capacity, but the new GPU pod still needs to start, obtain a device, load weights, and become ready.

Suppose the receipt model takes 42 seconds from activation to readiness and the endpoint promises a 200-millisecond response. Scale-to-zero is incompatible with that promise unless another warm layer absorbs the request. Keep at least one warm replica, maintain a warm pool, or route the first request through an asynchronous queue. For overnight batch work, where a 42-second delay is harmless, scale-to-zero may be exactly right.

Scale-down needs hysteresis: different behavior when adding capacity and removing it. If the service drops from eight replicas to two immediately after a burst, the next burst pays the cold-start cost again. A stabilization window and a conservative scale-down rate prevent the familiar sawtooth pattern of “scale out, shed traffic, scale in, suffer.”

Do not copy thresholds blindly. Derive them from a load test using production-shaped inputs. A text classifier, an image generator, and an LLM will have very different useful GPU targets. High utilization may improve cost per request for a well-batched model, but adding replicas can reduce batch sizes and make each request more expensive.

Failure modes you should recognize

The first symptom is rising p95 latency while CPU stays low. GPU utilization is high and the queue grows. This is the classic CPU-scaling failure. Move the primary signal to queue wait or backlog and expose accelerator metrics as a secondary signal.

The first symptom is replicas bouncing between two and eight. A delayed latency metric, an overly narrow target, or immediate scale-down is usually involved. Add a scale-down stabilization window, limit the rate of change, and prefer a leading queue signal.

The first symptom is an HPA status showing an unknown metric or an event such as failed to get pods metric. The application may be exporting to Prometheus correctly while the Kubernetes custom-metrics API has no matching metric, label, or adapter rule. Check the HPA events and query the metrics API itself. A graph in a dashboard proves that Prometheus has data; it does not prove that HPA can consume it.

The first symptom is a high queue with many Pending pods. The pod policy has requested more replicas than the GPU node pool can supply. Check node capacity, quotas, device-plugin health, and node-autoscaler time before tuning the HPA threshold.

What they’ll ask next

Why not scale directly on latency?

Latency is useful, but it is usually a lagging signal. Queue wait and in-flight work reveal pressure before the SLO is breached. I would use latency as a guardrail and validation metric, then tune the leading metric until the p95 and p99 targets remain acceptable.

How would you autoscale an LLM service?

I would separate prefill and decode pressure instead of relying on requests per second. I would monitor waiting requests, time to first token, inter-token latency, token throughput, and KV-cache occupancy. The right policy depends on whether the deployment is optimized for prompt processing, generation, or a disaggregated design where those stages scale independently.

Would you ever use CPU utilization?

Yes. If tokenization, feature extraction, image decoding, or post-processing is CPU-bound, CPU can be the correct signal. I would confirm that with profiling and load tests. The mistake is treating CPU as a universal proxy for inference load.

Say this in the interview

“Use queue wait or backlog as the leading signal tied to the latency SLO, add GPU or model-specific metrics to identify the bottleneck, expose them to HPA through custom metrics, and keep warm replicas whenever cold-start time cannot fit the endpoint’s SLA.”