Six system-design boundaries that prevent category mistakes
Stateless vs stateful, Lambda vs ECS, database vs cache, queue vs stream, retrieval vs reranking, and monitoring vs tracing—explained as operational contracts.
Most bad architecture diagrams are not missing a service. They have assigned the wrong responsibility to a service.
A cache has become the only copy of an order. A Lambda function relies on the previous warm invocation. A reranker is blamed for documents that retrieval never returned. A dashboard says latency is high, but nobody can follow a request far enough to find the delay.
Six common “A versus B” questions expose the same habit: choosing from product names before defining the contract.
State belongs somewhere, even in a stateless system
Stateless does not mean no database, no login and no chat history. It means one compute replica does not need its own prior request history to process the next request correctly. Put the session in a shared store and every API replica is replaceable.
Stateful components own evolving information that recovery must respect: database pages, stream offsets, workflow progress or a local index. Scaling now requires partition ownership, replication and restoration—not just a load balancer.
The test is simple: if this instance disappears, can another instance continue from durable shared information?
Lambda and ECS choose an execution model
Lambda is designed for short, event-driven invocations and scales per request. Its local environment is disposable and an invocation has a maximum duration. ECS schedules containers and is suited to long-running APIs, steady consumers, custom runtimes and explicit resource control.
That does not make ECS “the stateful option.” A container is replaceable too. Critical state belongs in a database, object store or replicated state system unless the service has an intentional recovery design.
Choose Lambda when events are brief and bursty. Choose ECS when the process must remain available, run longer, keep stable connections, or needs container-level control. Then make an independent state decision.
Database and cache choose authority
A database usually owns truth: durable writes, constraints, transactions and recovery. A cache owns speed under a staleness policy.
Cache-aside makes the relationship explicit: read Redis; on a miss, read the database and populate Redis with a TTL. The design is incomplete until it handles invalidation, stampedes, cache failure and stale values.
If deleting the cache destroys business truth, it was never merely a cache.
Queue and stream choose consumer semantics
A work queue coordinates ownership. A worker receives or leases a message, processes it, acknowledges success and retries failure. The natural question is “who will complete this job?”
A retained event stream coordinates independent views of history. Consumers track offsets and may replay events. The natural question is “which applications need to observe this event sequence?”
Services increasingly support features from both worlds, so brand names are weak evidence. Ask about retention, replay, ordering, acknowledgement, consumer groups and delivery guarantees.
Retrieval and reranking choose a stage objective
Retrieval cheaply reduces millions of candidates to perhaps 100. It should optimize recall: do not lose the relevant document.
Reranking spends more compute on those 100 and improves precision at the top. A cross-encoder can jointly inspect query and document because it no longer needs to scan the full corpus.
The first stage creates a hard ceiling. A reranker cannot promote a document it never received. Measure recall before asking the second stage to perform magic.
Monitoring and tracing choose investigation scope
Monitoring aggregates system health across time: request rate, failures, queue age, cost and latency percentiles. It detects that something is wrong.
Tracing records the path of one request through nested spans. It localizes where that request waited, retried or failed.
The useful workflow uses both: a metric alert detects a p95 regression, a trace locates the slow reranker call, logs explain its timeout, and metrics confirm the fix across the fleet.
The architecture is the composition
A production RAG service can be stateless at the API tier, use Lambda for upload events and ECS for long document processing, keep truth in a database and hot metadata in a cache, assign jobs through a queue, publish lifecycle events to a stream, retrieve broadly, rerank narrowly, monitor the fleet and trace individual requests.
These pairs are not mutually exclusive technology camps. They are distinct responsibilities that often coexist.
Work through the interactive System Design Boundaries lesson for decision tests, failure modes and deeper links.