Infrastructure
34 articles in this topic.
Six system-design boundaries that prevent category mistakes
Stateless vs stateful, Lambda vs ECS, database vs cache, queue vs stream, retrieval vs reranking, and monitoring vs tracing—explained as operational contracts.
How vLLM actually serves a 7B model
Follow one request through vLLM — the scheduler, the KV-cache blocks, prefill vs decode, and what happens when 90,000 tokens of cache no longer fit.
The vector-search memory wall: why HNSW eats RAM, and how quantization cuts the bill 32×
HNSW is fast because the whole graph lives in RAM — which is exactly why it gets expensive. At 100M vectors you're paying for ~600GB of memory before you serve a single query. Here's the math, and how binary quantization plus reranking is rewriting the cost model in 2026.
Don't auto-ship retrained models: collapse, feedback, and the challenger gate
Retraining can produce a worse model—from bad data, a pipeline bug, or learning from outcomes collected under its own serving decisions. Champion-challenger automates retraining while limiting the blast radius of regressions.
GPU compute isn't always the bottleneck: why decode can become a memory problem
During decode, LLM serving can depend on both HBM capacity and bandwidth: every step reads model weights and the relevant KV state, with KV traffic becoming especially important for long contexts and larger batches. Paged allocation reduces wasted capacity, and continuous batching can improve throughput and utilization, depending on the workload and its compute, bandwidth, communication, and latency limits.
Loading a model file can run code: MLSecOps in 2026
Downloading an untrusted pickle-backed artifact and loading it with an unrestricted or vulnerable deserializer can be as dangerous as running an untrusted script. Real malicious models have shipped on public hubs — here's the ML attack surface and the defenses that belong in your pipeline.
When much of your traffic is easy: use routing to cut LLM costs
When much of your LLM traffic is easy, route easy requests to a cheaper model, escalate uncertain cases, and cache safe repeats—but measure successful tasks, because a cheap wrong answer is not a saving.
Batch can be 100x cheaper for low-utilization, predictable workloads
The default mental model of 'serving a model' is a live API answering in milliseconds. For many predictable workloads, that is the expensive wrong choice — and the hybrid precompute-to-Redis pattern gives you batch economics with real-time lookup latency.
Many ML data failures are silent: the case for data contracts
Many ML data-quality failures stay silent when inputs remain type-compatible but their meaning changes. An enforced contract can block declared, testable violations at a chosen boundary before they poison retraining or serving; semantic guarantees require explicit producer-side checks against canonical data, paired representations, trusted baselines, or an equivalent mechanism.
Your GPUs are mostly idle: FinOps for the AI era
A practical guide to the cost of GPU allocation, useful work, and the fixes that turn idle accelerator time into completed runs and successful inferences.
For some AI deployments, deliverable electricity is becoming a binding co-constraint
For some AI deployments, deliverable electricity is becoming a binding co-constraint. Why data-center power matters, and how inference makes every AI query an energy cost.
Training-serving skew: the bug feature stores exist to kill
The model scores 0.95 offline and dies in production. Almost always the cause is one bug: training-serving skew, where the feature the model learned offline is computed differently online — or worse, was joined from the future. Here's the bug, why your holdout set can't catch it, and the three fixes that actually work.
MLOps is a loop, not a pipeline
Shipping a model is not the finish line — it is the starting gun for a feedback loop that runs as long as the model serves traffic, and the teams that forget this find out when a customer calls.
Denial of wallet: the attack that bankrupts your AI app
Classic DoS tries to take you down. Denial-of-wallet keeps you up and runs your token bill to five figures by lunch. Why request limits don't save you.
Surviving 10,000 concurrent requests to your LLM API
An LLM request spends 99% of its life waiting. Design around that one fact and 10k concurrent users stops being scary.
Cache the question, not just the bytes
Your embedding API answers the same query a thousand times a day. Three caches cut LLM cost and latency — and one of them can hand back a confidently wrong answer.
50,000 documents, one summarization pipeline
You can't loop 50k LLM calls inside a request. The shape that works — a queue, a pool of workers, and the boring reliability details that decide whether it finishes.
Where your Python packages actually live
venv isolates them, uv makes installing them almost free, and pex bundles them into one file you can throw at any machine. A field guide to three tools and the one question they each answer.
Edge AI in practice: Vercel AI SDK + Cloudflare Workers AI
Some inference belongs at the edge — the user's nearest POP — not in a central GPU cluster. The Vercel AI SDK and Cloudflare Workers AI made that practical. Here's where edge wins, how the cold-start tricks work, and what the streaming-from-the-edge architecture looks like.
Speculative decoding in the wild: how labs cut latency by 2-3x
A small fast model proposes, the big slow model verifies. Across Medusa, EAGLE, and draft-model approaches, speculative decoding is the latency-reduction lever that frontier labs reach for when they can't make the model itself any smaller.
Continuous batching: the one trick that made LLM serving 10x cheaper
Static batching wastes GPUs by padding to the longest sequence and waiting for stragglers. Continuous batching reschedules every iteration, evicts finished requests, and admits new ones mid-stream. Here's why that single shift bought a 10-23x throughput win.
Computer-use latency engineering: getting browser agents under a second
A naive browser-use loop is 4-8 seconds per step. Production systems run at 600ms. The gap is closed by half a dozen techniques — differential screenshots, prompt caching, batched actions, vision-model routing — each of which sounds boring until you measure the difference.
Prompt caching: the 90% cost cut explained
Anthropic, OpenAI, DeepSeek and Google all ship prompt caching now, but the pricing models diverge sharply. For a tool-using agent with a long system prompt, getting the cache pattern right is the difference between a viable product and a P&L disaster.
vLLM vs TGI vs SGLang: choosing your inference server in 2026
Three open-source serving stacks, three different bets. vLLM optimises raw throughput. TGI optimises ease and HuggingFace integration. SGLang optimises structured generation and prefix-cache reuse. Here's how to pick.
How Anthropic serves a hundred million tokens a second
The frontier labs and their serving partners — Anthropic, OpenAI, Together AI, Fireworks — hide an arsenal of inference optimisations behind the simple-looking chat endpoint. Here's the hierarchy, what each layer bought, and the public numbers behind it.
MLOps platform consolidation: Databricks, Snowflake AI, SageMaker
The MLOps landscape of 2023 — a dozen point tools, three feature stores, four monitoring vendors, two training frameworks — has collapsed. By 2026 the workload lives on Databricks, Snowflake Cortex, SageMaker Unified Studio, or Vertex. Here's what each platform actually does, what got swept under, and who wins which workload.
Feature stores in 2026: Tecton, Feast, Hopsworks — death and rebirth
The feature store hype cycle went peak (2021), trough (2023, 'just use dbt'), and into a quieter rebirth as the data-for-AI layer for both classical ML and LLM agents. Tecton sold to Databricks. Feast survives as the open-source baseline. Hopsworks redefined itself as an AI lakehouse. Here's what the modern feature store actually is in 2026.
The vector database shakeout: Pinecone, Weaviate, Qdrant, Chroma, pgvector
The pure-play vector DB category isn't dead, but it's smaller than the 2023 hype suggested. Postgres ate the low end, Pinecone owns serverless ops, Qdrant is the open-source cost winner, and Weaviate's hybrid search is its real moat. Here's the 2026 picture by the numbers.
Distributed training: FSDP vs DeepSpeed vs Megatron in production
For serious pretraining or fine-tuning, you pick from three: FSDP2 (PyTorch native), DeepSpeed (ZeRO stages), or Megatron-LM (NVIDIA, 3D parallel). The frontier labs have made their bets — Llama 3 went Megatron, FSDP2 is the open-source default under 70B, and DeepSpeed survives where ZeRO offload is necessary. Here's how to choose.
Mixture of experts in production: Mixtral, DeepSeek, Llama 4
MoE went from research curiosity in 2021 to production default in 2025. The new frontier-model template is sparse, with 5-15% active parameters per token. Here's how the serving infrastructure caught up, what vLLM and SGLang actually do with experts, and the cost math that makes MoE the future.
Quantization in production: GPTQ, AWQ, GGUF, FP8 — what to ship
16-bit serving is dead for most production workloads. Here's how to pick between weight-only post-training quantization, activation-aware quantization, and the FP8 native path — without measuring it wrong.
KV cache management: paged attention, prefix caching, LMCache
The KV cache is the dominant memory bottleneck in LLM serving, and three ideas — PagedAttention, prefix caching, and cross-instance LMCache — have rewired how it's managed. Here's how each layer earns its place in production.
NVIDIA Dynamo, vLLM, SGLang: serving stacks at scale
NVIDIA's Dynamo (GTC 2025) reframes LLM serving around prefill/decode disaggregation. vLLM, SGLang and TensorRT-LLM all sit underneath. Here's how the four-layer stack actually works, what the throughput numbers really mean, and who picks which.
Inference routing: sending each query to the cheapest model that can answer it
By 2026 the biggest lever on inference cost isn't quantisation or batching — it's deciding which model touches each query. Four routing patterns, three rounds of vendor consolidation, and a real case study where a customer support agent cut spend 80% with no measurable quality drop.