System Design Boundaries — State, Compute, Data & Signals

Many architecture pairs sound like product comparisons. They are better understood as contracts. A database promises durable authority. A cache promises faster access under an explicit staleness policy. A queue coordinates work ownership. A stream preserves an ordered event history for consumers. Here are the six pairs this lesson works through — read each not as “which is better” but as “which property forces the choice”:

1. Stateless versus stateful

A stateless compute instance does not require its own previous request history to handle the next request correctly. Any healthy replica can receive the call. The overall application can still have state in PostgreSQL, Redis, S3 or a workflow store.

A stateful component owns evolving information that participates in correctness: database pages, a stream processor’s window, a workflow cursor, a leader lease or a local index. Replacing it requires restoration, replay, replication or reassignment.

Question	Stateless compute	Stateful component
Can traffic move to any replica?	Usually yes	Only with state transfer or shared durable state
What happens on restart?	Reload configuration and continue	Recover checkpoint/log or rebuild state
Scaling concern	Load distribution	Partition ownership, consistency and rebalancing
Example	API reading sessions from DynamoDB	PostgreSQL primary or keyed stream aggregation

User login does not make a web server stateful if the session lives in a shared store and every replica can validate it. Conversely, a “stateless” worker that depends on an unreplicated in-memory cursor is stateful in the way that matters.

2. Lambda versus ECS

AWS Lambda runs functions in response to events. Each invocation must behave as if local memory and disk may disappear; durable state belongs in an external service. Lambda is strong for sporadic or highly variable work, native event sources and short handlers. The current maximum execution duration is 15 minutes.

Amazon ECS schedules containers. With Fargate, AWS manages the underlying servers; with EC2 launch type, you manage cluster instances. ECS fits long-lived APIs, steady consumers, custom runtimes, background services and workloads that need explicit CPU, memory or networking control.

The useful decision sequence is:

1. Does one unit of work comfortably finish inside Lambda’s runtime and package constraints?
2. Is it naturally event-driven and bursty enough that per-invocation scaling is valuable?
3. Does it require a continuously listening process, a custom sidecar, stable connection pool or long task? Prefer ECS.
4. Where does durable state live? Keep it external in either design unless you have built explicit recovery.

3. Database versus cache

The database is normally the system of record. It owns transactions, constraints, durable writes and the recovery log. A cache stores a derived or temporary copy optimized for access latency and load reduction.

In cache-aside:

read → cache hit → return
     → cache miss → read database → populate cache with TTL → return

The hard work is not calling Redis. It is defining staleness and failure:

• Which write invalidates or updates the cached key?
• How stale may the value be?
• What happens when many clients miss the same hot key—a cache stampede?
• Can the application fall back to the database when the cache is unavailable?
• Is negative caching safe for “not found” results?

If losing Redis loses the only copy of an order, Redis was being used as a database, regardless of what the diagram called it.

4. Queue versus stream

A queue usually represents work waiting for ownership. Competing workers receive messages, temporarily hide or lease them, acknowledge success, and retry failures. With Amazon SQS, a received message remains in the queue but is invisible until deletion or visibility-timeout expiry. Standard queues are at-least-once, so workers must be idempotent.

A stream represents an ordered sequence of events retained for a configured period. Consumers track offsets or checkpoints. Different consumer groups can independently read the same order events to build fraud signals, analytics and notifications. Retention enables replay and new projections.

Need	Prefer queue semantics	Prefer stream semantics
One job handled by one worker pool	Yes	Possible, but not the clearest default
Several applications independently read every event	Fan-out layer required	Natural with consumer groups/subscriptions
Replay last week after fixing code	Usually awkward	Core capability when retained
Per-job ack, retry and DLQ	Core capability	Usually implemented through consumer offsets and retry topics
Long-lived ordered history	No	Yes

Real products blur the categories—Redis Streams has consumer groups; Kafka can distribute records within a group. Choose from the contract: retention, replay, ordering key, delivery guarantee and consumer ownership.

5. Retrieval versus reranking

Retrieval scans a large corpus cheaply enough to produce a candidate set. Dense embeddings, BM25 or hybrid search aim for high recall@k: include the useful document somewhere in the top 50 or 100.

Reranking applies a richer model to only those candidates. A cross-encoder can jointly inspect the query and each document, often improving precision at the top positions, but it is too expensive to score millions of documents.

millions of chunks
      ↓ fast retriever, optimize recall
top 100 candidates
      ↓ cross-encoder / LLM reranker, optimize ordering
top 5 context chunks

Measure the stages independently. If the gold document is absent from the retrieved set, no reranker can recover it. If recall is strong but the best document sits at rank 40, reranking is the likely lever. End-to-end answer quality alone cannot tell those failures apart.

6. Monitoring versus tracing

Monitoring typically uses metrics and logs aggregated across time: request rate, error rate, p95 latency, queue age, GPU utilization and token cost. It answers whether a population is unhealthy and drives alerts.

Tracing follows one request through services as a tree of spans. It answers where this request spent time or failed: API gateway, retrieval, reranker, model call or database retry.

OpenTelemetry treats metrics, traces and logs as complementary telemetry signals. A common incident flow is:

1. A latency SLO monitor pages because p95 crossed the threshold.
2. Exemplars or a trace ID select a slow request.
3. The trace shows 2.4 seconds in reranker.call after three retries.
4. Logs explain the timeout; a metric verifies recovery across the fleet.

Tracing every request at full fidelity may be costly or sensitive. Sample intelligently, retain errors and high-latency traces, and redact prompts, credentials and personal data before export.

Put the pairs together

Consider a document-enrichment service:

• A stateless API accepts uploads and stores job state in a database.
• A Lambda handler validates small upload events; an ECS worker handles long OCR jobs.
• A queue assigns each document to one worker; a stream preserves document lifecycle events for analytics and audit consumers.
• A cache holds repeated metadata lookups but never owns the document truth.
• Retrieval finds 100 policy passages and reranking selects five.
• Monitoring detects rising latency; tracing locates the slow OCR call.

Good architecture is rarely “A instead of B everywhere.” Each pair operates at a different boundary and often appears in the same system.

In one breath

• Read each architecture pair as a contract, and choose by the property that forces it — not by which sounds better.
• Stateless vs stateful turns on who owns evolving state; Lambda vs ECS on bursty events vs a long-running process.
• Database vs cache is source-of-truth vs fast copy (if losing the cache loses the only copy, it was a database); queue vs stream is work ownership vs replayable ordered history.
• Retrieval vs reranking splits recall (find it) from precision (order it) — a reranker can’t recover a document retrieval never surfaced.
• Monitoring vs tracing is “is the fleet unhealthy?” vs “where did this request fail?” — and a real system uses both sides of every pair at once.

Quick check

Quick check

0/4

Q1An API stores sessions in DynamoDB and any replica can handle the next request. Is the API tier stateless?

No, because users have sessionsYes; durable state is external and replicas are interchangeableOnly if it uses LambdaOnly if it has no database

Check answer

Q2Which requirement most strongly favors a retained stream over a work queue?

One worker must resize one imageFailed jobs need a DLQSeveral independent applications must replay the same ordered eventsA request must return quickly

Check answer

Q3Retrieval recall@100 is poor. Will a stronger reranker fix the missing documents?

Yes, alwaysNo; reranking can only reorder candidates the retriever suppliedOnly with a cacheOnly on ECS

Check answer

Q4A dashboard reports high p95 latency. Which signal best localizes the slow component for one request?

A trace with nested spansA monthly cost reportA cache TTLA queue retention setting

Check answer

Sources and deeper lessons

• AWS decision guide: Fargate or Lambda
• Amazon SQS message lifecycle
• Redis cache-aside
• OpenTelemetry signals

Continue with Queues & batch pipelines, Advanced RAG and Observability & tracing.