Six system-design boundaries that prevent category mistakes

At 03:17, Northstar’s customer-support assistant starts giving the wrong return policy. The dashboard looks healthy: p95 latency is 1.8 seconds and the error rate is 0.1 percent. P95, the 95th percentile, means 95 percent of requests finish at or below that latency.

The policy changed at 01:00. Redis still has the old copy. The retriever also failed to return the new document, so the reranker never had a chance to use it.

The ingestion trace stops at the API because the upload path did not inject W3C Trace Context, the standard format for carrying trace identity, into the document-worker message. The worker’s spans are therefore not connected to the producer. The later customer question is a separate trace, so a durable document ID is needed to relate the two paths.

There are plenty of components in this system:

• an API
• Redis
• a database
• a vector index
• a queue
• a stream
• serverless functions
• containers
• a reranker
• metrics and traces

None of those nouns explains the incident.

The architecture failed because responsibilities crossed boundaries. A cache was treated as truth. Retrieval was judged by the reranker’s answer. A trace was expected to replace a fleet-level metric. The diagram was detailed. The contracts were missing.

That is the argument behind these six distinctions: do not choose between products first. Decide what must remain true when a process dies, data is stale, a message is repeated, a document is missing, or one request becomes slow. The product follows from that contract.

A boundary is a promise under failure

A system-design boundary separates two jobs that fail differently.

These jobs make different promises:

• A stateless API replica promises to handle a request without depending on correctness-critical history kept privately by its own process.
• A database promises durable authority for particular facts.
• A queue promises work coordination.
• A stream promises access to an event history.
• A retriever promises to find a broad candidate set.
• A reranker promises to order that set more carefully.
• Monitoring watches the population; tracing follows one member of it.

The services can be hosted by the same vendor. They can even run in the same process. The distinction still matters because recovery, scaling and measurement are different.

Ask the uncomfortable question at each boundary:

What happens when this component disappears, returns an old value, receives the same message twice, or never sees the required input?

That question is more useful than asking whether a service is “modern.”

1. Stateless versus stateful: where does the next request find history?

A stateless component does not need private, correctness-critical history from an earlier request to process a later request correctly. It may read plenty of state. The required prior state must be obtainable from shared external storage; it does not have to live in one replaceable process.

Suppose Northstar has three API replicas. A customer sends a question to replica A, which stores the conversation in memory. The next request goes to replica B after a deployment or a load-balancer decision. Replica B sees no conversation and gives an answer without the customer’s earlier details.

Store the conversation in a shared database, and either replica can continue. Store a short-lived copy in a cache only if losing it is acceptable.

A local cache is evolving state, but it can be safely disposable: a miss costs latency rather than correctness. The API remains stateless with respect to the conversation even though the overall application has state.

Durable acceptance and retry

For an asynchronous request whose retry must not create a second job, use idempotency, meaning a repeat has the same business effect as one attempt. Store a durable idempotency record with a uniqueness or conditional-write constraint scoped to the caller and operation.

Atomically claim the key and create the job record in the same transaction. This prevents two concurrent requests from both observing a missing key and creating jobs. On replay, return the existing job and its status.

Make downstream side effects idempotent too. Use the stable job ID so a retry updates the same logical work instead of creating a second result.

A stateless process can still lose an in-flight request when it dies. Statelessness does not mean crash recovery. The request’s prior state may be external while the response is still being computed in memory.

If the requirement is to resume work already accepted after a crash, that is a different contract. Record a durable job in a database or durable queue before acknowledging acceptance, then let another worker retry it.

A stateful component owns or is responsible for evolving information whose recovery matters. That information may be tied to a session, partition or service rather than one process.

A database owns pages and transactions. A stream consumer participates in maintaining a committed position, which may be stored by the broker or consumer group rather than in the worker process. A local index may own files that must be rebuilt or replicated.

Scaling such a component involves partition ownership, replication and restoration, not just adding instances.

The practical test is about a later request, not about resuming active work. After replica A disappears, can replica B obtain every correctness-critical fact needed to process the customer’s next request from shared external storage?

If yes, the API is stateless with respect to that fact. If no, the API has hidden session state.

Sticky sessions can make a broken design appear healthy by sending the same user back to the same replica. They do not solve a crash, a deployment or an exhausted machine. Warm memory is not a persistence layer.

2. Lambda versus ECS: how long and how deliberately must compute run?

AWS Lambda is a function execution service built around event-triggered invocations. Amazon ECS, the Elastic Container Service, schedules container tasks that can run continuously or process work for a long time.

Neither choice answers the state question.

When an administrator uploads a policy document, a Lambda function is a sensible place to validate the event, record a job and place work on a queue. The invocation is brief and bursty.

Lambda’s maximum single invocation timeout is 15 minutes. Its local filesystem and memory are not durable storage. A warm execution environment may be reused but is never a promise to your program.

The actual document pipeline may extract pages, call OCR, create embeddings and update an index. It may need a custom native library, a stable network connection or more than one short invocation.

An ECS worker consuming jobs is often the clearer execution model. It can run for hours, expose explicit CPU and memory settings, and be replaced without pretending that its memory is permanent.

The trade-off runs both ways. An always-running container has baseline cost and operational work when traffic is tiny. Lambda removes server management but brings invocation limits, startup effects, concurrency controls and event-specific retry behavior.

A small, short task does not become better because it was put in a container. A long task does not become serverless because it was split into hopeful callbacks.

Retryable jobs and index updates

Treat every job as retryable either way. Give every job a stable job ID and checkpoint progress, a saved marker of completed work, under that ID when resumption is safe.

For an index update, the safer default is to build a complete version in a staging index. Atomically swap the active alias—the stable name clients query—only after the build passes validation.

A retry can discard the incomplete staged version, resume from its checkpoint, or replace that version deterministically. If a worker writes half an index and then dies, the next attempt must never blindly append to the live index. It could publish a mixed or corrupt result.

The first symptom of a bad execution choice is often suspiciously precise. Jobs fail at exactly the configured timeout, or a retry creates duplicate embeddings because the first attempt completed the external call but died before recording completion.

That is an execution and idempotency problem, not a branding problem.

3. Database versus cache: which copy is allowed to be wrong?

A database usually owns the authoritative copy of a fact: the source of truth that recovery and business rules rely on. A cache is a faster, disposable copy whose freshness is governed by a policy.

For Northstar, the database might say that damaged items can be returned within 60 days. Redis may hold that policy for quick reads.

If deleting Redis makes the assistant slower but still correct, Redis is behaving as a cache. If deleting Redis erases orders or makes the company unable to reconstruct them, Redis was being used as a database whether the diagram admitted it or not.

The common cache-aside pattern makes the relationship visible:

1. Read the key from the cache.
2. On a miss, read the database.
3. Put the result in the cache with an expiry.
4. Return the result.

The pattern improves latency and reduces database load because repeated reads avoid the slower authoritative system. It does not solve freshness automatically.

Imagine a 120-second cache expiry. The database is updated at 01:00, but invalidation fails. A request at 01:01 can still receive the old policy.

Expiry limits the stale window in the uncomplicated case. It does not guarantee immediate freshness. An old read can also race with an update and repopulate a key after an invalidation.

Versioned values, careful write ordering or a write-through design may be needed when that race matters.

Expiry creates another operational edge. If 1,000 requests for the same popular key arrive just after it expires, all 1,000 can miss and hit the database unless the application coalesces requests or refreshes proactively. That is a cache stampede.

The database is not automatically the authority for every kind of data. The object store may own the original document, while the database owns its metadata and permissions. A vector index may be a derived search representation.

Write down the owner of each fact. “The database is the source of truth” is too vague to debug.

The first symptom of a category mistake is often a cache flush that causes a business outage, or a customer who sees an old policy only on some requests. The fix is to define allowed staleness and recovery, not to increase the cache size.

4. Queue versus stream: who needs the message, and for how long?

A work queue coordinates completion. A producer puts a job on it, a worker takes responsibility, processes the job and acknowledges success, sends the broker a success confirmation, according to the broker’s contract.

The natural question is: who will do this work?

A retained event stream provides replayable history. Its normal ordering guarantee is only within a defined partition or shard, often selected by a key. It does not imply one order across the entire stream.

A consumer or consumer group tracks a committed position, usually one position per partition, in that history. In a consumer-group system, those positions are commonly associated with the group and broker rather than with one particular consumer process. Another member can therefore resume after reassignment.

Global ordering across all records requires a single partition or additional coordination. A single partition limits parallelism and throughput.

Coordination across partitions preserves more parallelism but adds complexity and can hurt availability. There is no free total order hiding behind a product name.

The natural question for a stream is: which applications need to observe this fact?

Northstar can put “parse this document” on a queue. One worker should normally complete that job. If the worker crashes before acknowledging it, the broker can make the job available again.

The worker must tolerate that possibility. If processing takes 90 seconds but the queue’s temporary invisibility window, the period before an unacknowledged job becomes visible again, is 60 seconds, another worker may receive the same job while the first is still running.

Publishing committed facts

Northstar should not simply publish “document processed” after recording the durable result. A database commit can succeed before the process dies, losing the event.

Publication can also succeed before the process dies while marking the publication complete. That causes a retry to send the event again.

Use a transactional outbox, a pattern that commits the result and an event record in one database transaction. A relay reads the committed outbox row and publishes it to the stream with retries.

If the database commit succeeds and the process dies before publication, the relay still has the row to send. If publication succeeds and the relay dies before marking the row sent, it may publish again.

Consumers must therefore deduplicate by event ID or make their effects idempotent. Change data capture, or CDC, is another option: it emits events from committed database changes.

If the broker and database support a genuine shared transaction that atomically commits both, that is an alternative. Two ordinary systems do not make sequential writes atomic.

Search indexing, audit reporting and customer notifications can each consume the event. Separate consumer groups maintain separate committed positions and can replay old events after a bug fix. Members within one group share the work and partitions.

Products blur the line. A stream can distribute work. A queue can retain messages for a while. Retention alone does not tell you the intended contract.

Specify acknowledgement, replay, ordering scope, delivery guarantees and whether one or many independent consumers must see each event.

Use a queue when one action needs an owner and a retry. Use a stream when the event itself is valuable history.

A stream brings storage, partitioning and replay decisions. A queue is usually simpler when no independent reader needs the past.

The first symptom of using the wrong model is an audit consumer that misses documents because a job queue delivered them to the indexer. It may also be a worker that replays a six-month event history when it only needed to perform today’s task.

5. Retrieval versus reranking: find broadly, then order carefully

Retrieval is the first search stage that reduces a large corpus to a manageable candidate set. Reranking is a later stage that scores those candidates more expensively and decides which deserve the top positions.

This distinction is the difference between recall and precision.

Suppose Northstar has 10 million document chunks. Retrieval returns 100 candidates using keyword search, vector similarity or both.

On ten evaluation questions, the chunk needed to answer appears in nine of those candidate sets. For this simplified test, recall at 100 is 90 percent.

Even a perfect reranker cannot recover the missing chunk for the tenth question. If that chunk is necessary, the best possible end-to-end result for those questions is nine correct answers out of ten.

The reranker can improve the order of the nine successful candidate sets. It cannot promote a document it never received.

That is why reranking is valuable when the right document is present but buried at position 70. A more expensive model can inspect the query and candidate together and move the useful passage into the top five.

The cost is paid only for 100 candidates rather than 10 million documents.

A reranker can also make a good candidate list worse. Its training data may favor polished but generic passages or misunderstand negation.

It is not an authorization system. Tenant filters, which keep one customer’s data out of another’s, and ACLs, access-control lists that specify who may read each document, must be enforced before candidates reach the reranker where possible. They must also be rechecked before generation or display.

Letting an unauthorized candidate into the model can leak its contents through a generated answer even if the reranker gives it a low score. Ranking quality must never be used as permission enforcement.

Reranking adds latency and compute cost. On a small corpus, or when the first-stage results are already precise, it may not earn its keep.

Measure the stages separately. Record candidate recall at the chosen cutoff, then top-five answer quality, then reranker latency and cost.

The RAG basics and RAG evaluation lessons go deeper on those measurements.

The first symptom of blaming the wrong stage is a confident answer that remains wrong after the reranker is tuned. Inspect the candidate set first.

If the relevant passage is absent, change one or more of these:

• ingestion
• permissions
• chunking
• filters
• retrieval

Do not ask the final stage to perform archaeology.

6. Monitoring versus tracing: is the problem widespread or specific?

Monitoring aggregates telemetry, data emitted by a system, across a service or fleet. It answers whether request volume, error rate, queue age, cost or latency is changing.

Tracing follows one request through its operations. A trace contains spans, timed records for steps such as API handling, retrieval, database access and reranking. It answers where this particular request waited, retried or failed.

Averages hide tails. Imagine 1,000 requests: 990 finish in 200 milliseconds and 10 take 5 seconds.

The mean is 248 milliseconds, which sounds fine until the slow customer is on the phone. A percentile shows the shape of that experience. P95 means the point at which 95 percent of observed requests are at or below the measured time.

A fleet metric might alert when p95 rises from 1.2 seconds to 2.8 seconds. A trace for one slow request might show 140 milliseconds in retrieval, 1.9 seconds in the reranker and 430 milliseconds waiting for a database connection.

Logs can explain the timeout or input, while the metric tells you whether the repair helped everyone.

Neither signal replaces the other. Traces are often sampled to control storage cost, so a missing trace does not prove that no slow request occurred.

Metrics can show that the fleet is sick but cannot usually explain which downstream call caused the symptom.

Crossing service and queue boundaries

Context propagation must cross service and queue boundaries deliberately.

For an HTTP call, the producer injects W3C Trace Context into headers. The downstream service extracts it before creating its span.

For a queue, inject that context into message attributes. The worker extracts it and creates a consumer span linked to the producer span.

Asynchronous consumption is not automatically a continuation of the original request. The upload may finish long before a worker runs, and the original trace may already be closed.

The link records the relationship without pretending that the upload waited for the worker.

A common first symptom is an alert that says “latency high” with no route to investigate, or a trace that ends at the API even though the time was spent in a worker.

Add a request ID for searchable logs. Give each job or document a durable ID that appears in logs and message attributes.

Use that durable ID to relate separate traces when an upload and a later customer request are not one trace. A correlation ID is searchable metadata; it does not by itself connect OpenTelemetry, the instrumentation standard used to create and export these spans, traces.

Propagate trace context deliberately and measure each important boundary. The LLM inference metrics and ML observability guides cover the details that matter once model calls enter the path.

The strongest objection: modern products do all of these things

The objection is fair. Redis can persist data. A streaming system can distribute work. Lambda can start asynchronous workflows. ECS can hold state in memory.

One observability platform can show metrics, logs and traces on one screen. Strict categories can sound like old architecture advice in a world of managed services.

The answer is that capability is not the same as contract.

A persisted cache may still be rebuildable and therefore not authoritative. A stream used as a queue may retain events that nobody intends to replay.

A container with local state may work until the scheduler replaces it. A unified telemetry product still contains different signals with different questions.

These boundaries do not require six separate services. They require six explicit decisions.

One process can retrieve and rerank. One database can store job records and conversations. A single vendor can provide the queue and stream.

Keep the boundary in the design even when the boxes are combined.

What to do on Monday morning

Pick one real path, not the whole platform. Northstar could choose “customer asks a return-policy question” and “administrator uploads a new policy document.”

For each path, write five short contracts:

• Which component owns each fact, and what loss or staleness is acceptable?
• How long may each compute step run, and what happens when it is retried?
• Does each message need one worker, or must independent readers replay it?
• What candidate-recall and top-result measures prove that search works?
• Which metric detects a fleet problem, and which trace explains one request?

Then run failure drills before choosing another product. Use these scenarios:

• Kill an API replica during a conversation.
• Delete the cache.
• Deliver the same document job twice.
• Make a worker exceed its timeout.
• Start an audit consumer after events have accumulated.
• Remove the relevant document from the retrieval candidate set and verify that evaluation reports a retrieval failure rather than blaming ranking.
• Follow a trace across the queue: verify that the message carries W3C trace context, the worker creates a linked consumer span, and a durable job or document ID lets you find separate later traces.

Put the resulting measurements on a small dashboard:

• cache hit rate and stale-read errors
• queue age and retry count
• candidate recall
• reranker latency
• answer quality
• cost per request
• trace completeness

These are not decorative dashboard tiles. Each one corresponds to a contract you can test.

The final Northstar design may be quite ordinary: replaceable API replicas, durable conversation and document metadata, a cache with an explicit freshness policy, a short Lambda upload handler, ECS workers for long processing, a queue for jobs, a stream for lifecycle history, broad retrieval followed by narrow reranking, fleet metrics and request traces.

That design is not good because it contains the right brands. It is good because every boundary has an answer when the 3 a.m. failure arrives.

Sources

• AWS: Fargate or Lambda
• Amazon SQS message lifecycle
• Redis cache-aside
• OpenTelemetry signals

Check your understanding