A multi-agent system produces confident but contradictory reports, repeats work, and sometimes reaches a wrong consensus. How would you determine whether the problem is decomposition, communication, incentives, aggregation, or evaluation, and when would you replace it with a single agent?

How to think about it

I would first trace one bad case from task assignment through messages, tool calls, intermediate reports, aggregation, and final evaluation, then run controlled ablations against a single-agent baseline. I would replace the multi-agent system when the task is tightly coupled, the agents share the same failure modes, and coordination adds more latency, cost, and opportunities for error than it removes.

Why this is the real problem

“Consensus” means agreement. It does not mean truth.

A majority vote helps only when the agents’ errors are at least partly independent. If four agents read the same stale document, use the same model, and inherit the same misleading premise, they are four witnesses to one mistake. Their agreement is evidence of correlation, not correctness.

Confidence needs the same suspicion. A language model’s statement that it is “95% confident” is usually generated text unless the system has calibrated that confidence against labeled outcomes. Calibration means that cases assigned 80% confidence are correct about 80% of the time. Fluent certainty is not calibration.

I would measure more than final accuracy:

• Correctness: Did the final answer match a trusted outcome?
• Calibration: Did confidence track correctness?
• Contradiction rate: How often did reports disagree on material facts?
• Duplication: What fraction of retrievals, tool calls, or reasoning steps repeated the same work?
• Cost and latency: What did the system spend to reach the answer?
• Coverage: How often did it abstain when evidence was insufficient?

Then I would create a labeled replay set. For an incident-analysis system, that might be 50 historical incidents with a postmortem-approved root cause and the evidence that was available at decision time. I would freeze the set. Otherwise, a clever evaluator can quietly turn yesterday’s answer into today’s “ground truth.”

The key is to change one thing at a time. Remove communication but keep the task split. Keep communication but replace majority voting. Keep the prompts but remove any reward for agreement. This turns a vague “the agents are bad” diagnosis into a causal one.

A concrete diagnosis

Consider a 3 a.m. checkout outage. The system has four investigators and one synthesizer:

• a deployment investigator
• a database investigator
• a traffic investigator
• a security investigator

The checkout error rate rises from 0.4 percent to 18 percent at 02:14. The deployment investigator reports a newly released payment service as the cause with 90 percent confidence. The database investigator reports exhausted connection slots with 85 percent confidence. The traffic investigator repeats the deployment theory with 95 percent confidence. The security investigator repeats the database theory with 80 percent confidence.

The synthesizer chooses “bad deployment” by majority vote. The postmortem later shows the cause was a traffic failover that sent too many requests to a database whose connection limit was already low. The deployment was unrelated.

I would inspect the trace before rewriting any prompt.

Suspected locus	Controlled test	Diagnostic signal
Decomposition	Give one agent the whole case, then compare with the proposed subtasks	A subtask requires facts assigned to another subtask, or several tasks overlap
Communication	Replay with no messages, then with structured evidence and source identifiers	The needed fact exists but never reaches the agent that needs it
Incentives	Remove agreement rewards and add a cost for duplicate retrievals	Agents stop copying one another or produce useful dissent
Aggregation	Compare majority vote with evidence-weighted adjudication and abstention	Correct reports are discarded because weak, repeated reports outvote them
Evaluation	Score against the postmortem and inspect the judge rubric separately	The system is rewarded for confidence, agreement, or persuasive prose rather than truth

Suppose the trace shows all four investigators queried the same 500-line log excerpt. Three copied the same sentence about the deployment. None saw the database connection metric because the task router assigned it only to the database investigator, and that report was truncated before synthesis.

That is not one failure. It is a chain:

1. Decomposition failure: “Investigate the database” was too narrow; the database investigator needed traffic and failover context.
2. Communication failure: the synthesizer received conclusions but not the metric and source identifiers supporting them.
3. Aggregation failure: three similar claims beat one directly measured claim.
4. Evaluation failure: the judge rewarded a decisive answer instead of a sourced, uncertainty-aware one.

The experiment should confirm each link. Give every investigator the relevant evidence, require each claim to carry a source_id, and replace majority voting with a rule that prefers directly observed evidence while allowing “insufficient evidence.” If accuracy improves, contradiction falls, and duplicate retrievals fall from, say, eight calls to three, the trace has identified a mechanism rather than merely producing a nicer answer.

The senior-level nuance

Incentives do not require a reinforcement-learning reward. A prompt that says “produce a definitive consensus,” a judge that scores concise agreement highly, or a workflow that gives no penalty for repeated searches all create incentives. The system will optimize what is measured, including the wrong thing.

Likewise, different job titles do not create genuine diversity. Four copies of the same model with prompts saying “you are a database expert” may still share the same training blind spots and retrieve the same source. Useful diversity usually comes from different evidence, tools, models, or methods. One agent might inspect metrics, another deployment history, and a third run an executable health check. Even then, the final decision needs provenance, not just votes.

A common failure mode appears first as a polished report containing four near-identical paragraphs and no source identifiers. The system looks coherent because the synthesizer has compressed the disagreement away. Another appears as an empty or generic final answer after several rounds: context limits have removed the original evidence, leaving agents to reason over summaries of summaries. Logging token counts, message sizes, and which evidence survives each round catches this quickly.

I would not assume that a clever aggregator fixes bad decomposition. If the subtasks are not separable, the agents may each be locally correct while the system is globally wrong. Root-cause analysis, legal reasoning, and architecture design often have dependencies that emerge only when evidence is combined. For those tasks, a single planner with a shared working set may outperform a committee of isolated specialists.

The opposite is also true. If four independent documents must be checked, or four slow tools can run in parallel, specialization can be worthwhile. A four-agent design that completes four independent checks in 30 seconds may beat one agent doing them sequentially in two minutes, even after a 20-second synthesis step. But if each agent needs three rounds of discussion and the final answer is only a short lookup, the committee is ceremony with a token bill.

When I would replace it with one agent

I would use a single agent when:

• the task has one tightly coupled context rather than cleanly separable subtasks;
• agents repeatedly consult the same evidence;
• their errors are correlated;
• the synthesizer cannot verify claims independently;
• coordination consumes most of the latency or token budget;
• a single agent matches or beats the committee on a held-out, labeled set;
• deterministic validators or external tools can check the important outputs.

“Single agent” does not mean “one unstructured prompt.” It can still plan, call tools, maintain a scratchpad, cite evidence, and run a final verification pass. The distinction is that one controller owns the context and the decision, instead of negotiating with several peers whose partial views may be incompatible.

I would retain multiple agents when the work is genuinely parallel, evidence sources are independent, specialists have different capabilities, and the outputs can be checked mechanically or by a clearly defined adjudicator. I would also compare the total system, not just answer quality: a one-percentage-point accuracy gain is hard to defend if it costs five times as much and doubles the 3 a.m. response latency.

What they’ll ask next

How do you measure whether confidence is useful?
Use a held-out labeled set. Group predictions by confidence, compare stated confidence with actual accuracy, and track calibration alongside accuracy. Also allow abstention; a system that says “the evidence is insufficient” can be safer than one that always chooses a winner.

How would you create agent diversity?
Vary the evidence and method before varying the personality prompt. Use independent retrieval paths, different tools, or genuinely different models where justified. Then measure error overlap. If two agents are wrong on the same 30 of 50 cases, calling them independent is wishful bookkeeping.

What if there is no immediate ground truth?
Use executable checks, later operational outcomes, expert adjudication, or delayed labels. Separate “we cannot verify this yet” from “the agents agreed.” Agreement is not a substitute for evaluation.

The line I would use in the room is: “I would debug the causal chain from evidence to decision, not reward the system for sounding unanimous; if one agent with tools and validation is as accurate with half the coordination, I would ship the simpler design.”