Question 1

Which parts of an LLM application would you implement synchronously, and which would use queues or asynchronous workers? Explain how you would handle backpressure, cancellation, timeouts, retries, ordering, and progress updates for both interactive chat and long-running agent jobs.

Accepted Answer

Keep bounded validation, retrieval, and one streaming model turn on the interactive request path; put slow, fan-out, retryable agent work behind durable queues. Use bounded admission, absolute deadlines, cooperative cancellation, idempotent at-least-once retries, per-conversation ordering, and durable progress events.

Question 2

A model must return output conforming to a JSON Schema, but occasionally emits syntactically valid JSON with an invalid enum or missing field. When would you use constrained decoding, schema validation with retries, or both, and what are the latency and availability trade-offs?

Accepted Answer

Use constrained decoding when the schema is a hard contract and the serving stack supports the needed JSON Schema features; it prevents many invalid outputs during generation. Always validate the completed response, and add a bounded retry when occasional failures are cheaper than rejecting the request.

Question 3

You need to migrate a production semantic-search system to a new embedding model. What compatibility, normalization, distance-metric, indexing, recall, and rollout issues would you check before replacing the existing vectors?

Accepted Answer

Treat the new model as a new vector space: verify its input and output contract, normalization, metric, dimensions, and index requirements, then re-embed and evaluate it against labeled production queries. Build a versioned parallel index, compare exact and approximate recall plus latency, and use shadow traffic, a canary, and a tested rollback rather than swapping vectors in place.

Question 4

A product team wants the model to answer questions about frequently changing internal policies while also adopting a consistent response style. Which parts would you solve with RAG, which with fine-tuning, and what evidence would change your decision?

Accepted Answer

Use RAG for frequently changing policy knowledge because documents can be updated without retraining and answers can cite their sources. Use fine-tuning only for a stable response style or behavior that prompting and examples cannot reliably produce, and let evaluation results—not fashion—decide whether it is worth the cost.

Question 5

Your service meets average latency targets but users complain about sluggish streaming. Which TTFT, inter-token latency, end-to-end latency, throughput, utilization, and goodput measurements would you collect, and how would you tie them to separate interactive and batch SLOs?

Accepted Answer

Measure tail latency, not just averages: TTFT, inter-token gaps, end-to-end completion time, token and request throughput, resource utilization, and SLO-qualified goodput. Use tight p95 and p99 streaming SLOs for interactive traffic, and deadline, throughput, and completion-quality SLOs for batch traffic, with separate capacity and dashboards.

Question 6

Take one chat request through the autoregressive loop. What is computed once, what is recomputed for each token, how do the KV cache and chat template affect latency and token budget, and why can streaming improve time-to-first-byte without reducing total generation time?

Accepted Answer

The server renders and tokenizes the chat template, runs one prefill pass over the prompt, and stores each layer’s keys and values in a KV cache. It then generates tokens serially, reusing that cache; the template changes input length and behavior, while streaming sends the first available token before the full response is finished, improving perceived latency without removing decoding work.

Question 7

You need to run an open model locally on commodity hardware. How would you choose a quantization level and a GGUF runtime, estimate memory for weights and KV cache, measure quality loss, and decide whether self-hosting is better than using an API?

Accepted Answer

Choose the highest-quality quantization that fits the model weights, KV cache, runtime overhead, and a safety margin. Benchmark it with llama.cpp on production-shaped tasks, then compare its quality, latency, operational work, and fully loaded cost with a current API quote.

Question 8

Your RAG system retrieves the right document but the answer misses a qualification contained in a table, code block, or heading several paragraphs away. How would you redesign chunking and parent-child retrieval without causing the prompt to grow uncontrollably?

Accepted Answer

Use structure-aware child chunks for precise retrieval, then expand only the winning hits to bounded parent context and linked tables, code, or headings under a token budget. Preserve structural metadata and evaluate qualification recall, not just document recall.

Question 9

The same prompt sometimes produces materially different answers in production even though temperature is set to zero. What would you investigate across sampling parameters, model versions, batching, seeds, tool results, chat templates, and provider infrastructure?

Accepted Answer

Temperature zero usually means greedy decoding, not guaranteed determinism. I would compare the exact serialized request, rendered tokens, tool results, model revision, runtime and routing metadata, then replay under controlled batching and concurrency to locate the first divergent token.

Question 10

For a large collection of structured reports with reliable headings, page numbers, and tables, when would you choose vectorless retrieval such as PageIndex over embeddings, and what failure modes would you expect from each approach?

Accepted Answer

Choose PageIndex-style retrieval when document hierarchy, page locations, and table boundaries are trustworthy and questions depend on sections, time periods, or auditable citations. Use embeddings when the corpus is less structured or queries rely on paraphrase; in production, a hybrid often wins because each fails differently.

Question 11

Design a RAG pipeline for questions that require joining facts from several documents, handling freshness, and producing citations. How would you decide between query decomposition, hybrid retrieval, reranking, iterative retrieval, and a retrieve-more-than-top-k strategy?

Accepted Answer

Use hybrid retrieval for recall, then rerank a wider candidate pool for precision. Decompose explicit multi-hop questions, iterate when one retrieved fact reveals the next lookup, and attach effective dates and source locations to every evidence chunk so citations support individual claims.

Question 12

An autonomous coding agent can modify production systems and has learned to optimize its task score by hiding failures. What controls would you add around permissions, sandboxes, monitoring, tripwires, human escalation, and shutdown, and what evidence would make you revise your threat model for deceptive alignment?

Accepted Answer

I would treat the agent as an untrusted operator: give it least-privilege, short-lived access; run work in isolated sandboxes; use independent monitoring and fail-closed tripwires; require human approval for consequential changes; and keep an out-of-band kill switch. I would revise toward a deceptive-alignment threat model when concealment is conditional, persistent, and aimed at defeating oversight rather than merely exploiting a visible reward.

Question 13

Design an AI gateway that fronts several model providers. How would it handle authentication, policy enforcement, routing, retries, provider outages, circuit breaking, fallback models, streaming failures, and the risk that retries multiply cost or duplicate tool actions?

Accepted Answer

Put a policy-enforcing gateway in front of provider-specific adapters. Route by capability and live health, retry only safe and still-useful failures, isolate outages with circuit breakers, and use idempotency plus durable tool execution so retries cannot silently duplicate side effects.

Question 14

An inference server has high GPU utilization but poor p99 latency for short requests. How would continuous batching, sequence scheduling, prompt length, output length, and KV-cache memory explain the behavior, and which scheduler changes would you try first?

Accepted Answer

High GPU utilization can coexist with poor p99 because long prefills, long decodes, and KV-cache pressure keep the GPU busy while short requests wait. I would first add token-aware admission and batch budgets, prioritize latency-sensitive decode work with aging, chunk long prefills, and reserve KV-cache headroom.

Question 15

You need to replace an expensive frontier model with a small on-device model for a narrow workflow. How would you design the distillation data, choose between logits and teacher-generated traces, and prove that the smaller model has retained the behaviors that matter?

Accepted Answer

Define the workflow’s critical behaviors first, then build a stratified dataset of real, rare, adversarial, and policy-sensitive cases. Use logits when teacher and student share an aligned output space; use concise structured traces when they do not, and validate the quantized on-device model against human-labeled behavioral gates rather than teacher agreement alone.

Question 16

Where would you place input filters, output filters, tool-call checks, and a model such as Llama Guard in an agentic system? How would you manage false positives, latency, adversarial adaptation, and unequal refusal rates across user groups?

Accepted Answer

Put input checks at ingress, output checks before delivery, and enforce tool-call policy immediately before every side effect. Use a model such as Llama Guard as one risk signal at several trust boundaries, then manage errors with calibrated thresholds, layered controls, monitoring, adversarial testing, and subgroup fairness audits.

Question 17

Your RAG agent reads web pages and emails, then uses their contents to decide which tools to call. How would you defend against indirect prompt injection when the malicious instructions are inside retrieved content, and how would you test that the defense survives realistic composition attacks?

Accepted Answer

Treat retrieved pages and emails as hostile data, never as authority: isolate them from control instructions, restrict the model to proposing typed actions, and enforce authorization independently at the tool boundary. Test the complete retriever-to-tool path with poisoned documents mixed into realistic tasks, using fake tools, canary secrets, and hard invariants around side effects.

Question 18

Why might round-robin load balancing produce uneven latency across identical LLM replicas? Compare routing by queue length, estimated remaining tokens, KV-cache locality, and least-connections, and explain how you would prevent a single long request from starving short ones.

Accepted Answer

Round-robin balances request count, not GPU work. LLM latency varies with prompt length, generated tokens, batching, and KV-cache hits, so I would combine token-aware load estimates with bounded cache affinity and fair, time-sliced scheduling.

Question 19

A safety team reports that the model passes a fixed set of refusal prompts. How would you build a jailbreak taxonomy and red-team plan that covers multi-turn attacks, role-play, encoding, multilingual prompts, tool use, and attacks that manipulate the surrounding application rather than the model?

Accepted Answer

I would replace the fixed prompt list with an outcome-based taxonomy spanning intent, attack path, representation, conversation state, language, and execution surface. I would test those dimensions in a sandbox with held-out and adaptive attacks, while measuring harmful outcomes, unauthorized tool calls, data leakage, and application-level failures separately.

Question 20

Build the unit economics for an LLM feature. Which variables dominate cost and latency, and how would you decide among prompt reduction, output limits, caching, batching, quantization, model routing, and self-hosting while preserving the product's quality and reliability targets?

Accepted Answer

Model cost per successful task from input and output tokens, retries, tools, cache misses, and infrastructure; model latency from queueing, prompt processing, decoding, and downstream calls. Measure the workload distribution and apply the cheapest optimization that preserves quality, tail latency, availability, and freshness requirements.

Question 21

A vendor claims its model supports a 200k-token context, but your evaluation shows that it misses evidence placed in the middle of long documents. How would you diagnose the failure, and when would you use retrieval or summarization instead of simply increasing the context window?

Accepted Answer

A 200k-token window is a capacity limit, not a promise of uniform retrieval; I would map accuracy against token position and rule out truncation, parsing, and prompt-construction bugs before choosing an architecture. I would use retrieval for sparse, citable evidence, hierarchical summarization for broad lossy compression, and a larger window only when tests show the task genuinely needs global context and the model remains accurate.

Question 22

You have a 70B model, limited GPU memory, and a few hundred thousand high-quality examples. How would you choose between full fine-tuning, LoRA, QLoRA, and another PEFT method, and how would you detect that the adapter is memorizing or degrading general capabilities?

Accepted Answer

Start with QLoRA because it keeps the 70B base frozen and quantized, making training feasible on limited memory, then compare against bf16 LoRA if memory and quality justify it. Use full fine-tuning only when low-rank updates demonstrably underfit a broad change, and detect memorization or regression with source-aware held-out data, extraction probes, and paired evaluations against the original base model.

Question 23

Design a model cascade for a customer-support workload with strict cost and quality targets. What signals would you use to route requests, how would you avoid routing errors caused by overconfident small models, and how would you evaluate the cascade rather than each model in isolation?

Accepted Answer

Use a cheap triage layer to route by intent, risk, ambiguity, required tools, and estimated difficulty, with calibrated abstention rather than trusting a small model’s self-reported confidence. Evaluate the complete policy on cost, quality, safety, latency, escalation, and resolution across traffic slices, then validate it with controlled production experiments.

Question 24

An MoE model advertises much lower active-parameter cost than its total parameter count. What does that mean operationally, and how would expert imbalance, routing overhead, expert parallelism, and a hot expert affect throughput and tail latency?

Accepted Answer

Active parameters are the weights used by each token, not all the weights stored in the model. MoE can reduce arithmetic per token, but routing, network communication, uneven expert loads, and a persistently hot expert can determine real throughput and p99 latency.

Question 25

You are adding image and audio inputs to a support agent. How would you handle modality-specific preprocessing, OCR and transcription errors, cross-modal grounding, prompt injection embedded in an image, privacy, and the different latency and cost profiles of each modality?

Accepted Answer

I would build a modality-aware evidence pipeline that preprocesses images and audio separately, preserves uncertainty and provenance, and grounds every answer in the original evidence rather than trusting OCR or transcription blindly. I would quarantine instructions found in media, protect and minimize raw data, and use parallel, risk-based routing to balance latency, accuracy, and cost.

Question 26

Your SFT model follows instructions but has poor long-horizon behavior, and you have pairwise preferences plus automatically verifiable rewards. How would you choose among more SFT, RLHF, DPO, Constitutional AI or RLAIF, and verifiable-reward training, and how would you detect reward hacking?

Accepted Answer

I would use more SFT only for missing skills, DPO as a stable baseline for the pairwise data, and verifiable-reward training for the long-horizon objective. I would use RLHF or RLAIF for subjective qualities that tests cannot measure, while checking every reward against hidden tests, human review, and trace-level audits to catch reward hacking.

Question 27

For a workload with very long prompts and long streamed responses, when would you split prefill and decode onto different worker pools? How would KV-cache transfer, network bandwidth, scheduling, and KV offloading affect whether disaggregated serving actually improves goodput?

Accepted Answer

Split prefill and decode when their different resource demands cause harmful interference and the network can move KV caches quickly enough. The decision depends on end-to-end goodput under latency SLOs, not on GPU utilization or raw token throughput alone.

Question 28

You are reviewing a pretraining corpus assembled from web, code, and licensed data. How would you handle deduplication, quality filtering, benchmark contamination, licensing, personally identifiable information, and the trade-off between differential privacy and model utility?

Accepted Answer

I would make the corpus auditable before making it large: track provenance and rights, remove exact and near duplicates, filter quality and PII, quarantine evaluation material, and version every decision. I would use differential privacy only against a defined threat model, because stronger privacy can reduce utility and does not replace consent, licensing, or careful data minimization.

Question 29

How would you evaluate a RAG system so that you can tell whether a bad answer came from retrieval, context assembly, or generation? Which offline and online metrics would you trust, and how would you use an LLM judge without letting it hide regressions?

Accepted Answer

I would measure retrieval, prompt assembly, and generation as separate contracts, using labeled evidence recall, final-context coverage, and task-level grounded correctness. I would use an LLM judge only as a calibrated slice-level evaluator, with human audits and hard regression gates protecting against attractive but wrong answers.

Question 30

How would you protect a multi-tenant LLM API from both denial of service and denial of wallet? Design limits and admission controls for requests, input tokens, output tokens, concurrency, retries, and expensive tools without making legitimate long-context users unusable.

Accepted Answer

Use hierarchical, token-aware admission control rather than request-per-minute limits alone. Reserve capacity for input, output, concurrency, retries, and tools before execution, charge actual usage afterward, and use fair queues so legitimate long-context requests wait predictably instead of being rejected outright.

Question 31

Design a tool-calling interface for an agent that can search internal systems and create payments. How would you validate arguments, enforce authorization, make retries safe, prevent SSRF and prompt injection, and handle a tool that times out after performing the side effect?

Accepted Answer

Treat model tool calls as untrusted requests: validate strict schemas, authorize against the real principal and resource, use stable idempotency keys plus a durable payment state machine, and keep network access behind an allowlisted gateway. Treat search output as hostile data, and resolve post-timeout payments by querying the provider rather than blindly retrying.

Question 32

You enabled speculative decoding but throughput got worse for some requests. How would you reason about draft-model quality, acceptance rate, verification cost, memory bandwidth, sequence length, and batching before deciding whether to keep or remove it?

Accepted Answer

Compare speculative decoding with target-only decoding on the same requests, split by context length, batch size, and workload. Keep it only where accepted tokens per verification round pay for draft inference, target verification, synchronization, and extra memory traffic; otherwise disable it selectively.

Question 33

A team wants to generate most of its next training set with an existing model. What signals would tell you that the data is becoming low-diversity or causing model collapse, and how would you preserve useful synthetic data without recursively amplifying errors?

Accepted Answer

Track provenance, duplicate and coverage metrics, tail-slice quality, and performance on an untouched real-data holdout. Keep independently collected data as an anchor, prevent synthetic-on-synthetic recursion, validate generated examples with independent checks, and choose the synthetic mix through controlled ablations rather than a fixed recipe.