Theory of mind for agents

It is 3:07 a.m. A monitoring agent sees 120 payment errors per minute and pages the on-call engineer. A deployment agent knows the service was rolled back three minutes ago. It expects recovery, but the monitor’s last rate measurement is still 120 per minute.

At 03:04, the deployment agent sent the monitoring agent only a signed rollback event and requested a later measurement. The event established that the service changed to version v41; it did not provide a fresh error-rate measurement.

The deployment agent replies, “The incident is fixed.”

The monitoring agent receives the reply and pages again. Its policy accepts only evidence in one of two forms: a fresh monitoring measurement for the error-rate alert or a signed deployment event for deployment state. “The incident is fixed” is neither a fresh measurement nor a signed event, and the earlier signed event did not establish recovery. The monitor ignores the unsupported claim.

Nothing is wrong with the monitoring agent’s arithmetic. It is acting on an older view of the service’s health. It saw the rollback event, but it had not yet seen a fresh measurement showing that errors had fallen. The deployment agent answered from its view of the deployment; the monitoring agent acted from its own evidence.

A useful multi-agent system must keep these separate:

• what is true,
• what each agent believes is true,
• what each agent wants,
• and what each agent has had a chance to observe.

The ability to represent another agent’s mental state is theory of mind, usually shortened to ToM. “Mental state” here means a belief, goal, intention, uncertainty, or knowledge state attributed to another agent. It does not require consciousness. It requires perspective.

That distinction is the difference between coordination and several bots shouting accurate facts into the same confused room.

The false-belief test

The classic probe for ToM is the false-belief test: one person has information another person lacks.

Sally puts a ball in a basket and leaves the room. While she is away, Anne moves the ball to a box. Sally returns. Where will Sally look first?

The ball is really in the box. Sally believes it is in the basket because the last location she observed was the basket. A system that answers “box” has read the world correctly but failed the perspective-taking task. A system that answers “basket” predicts Sally’s action from Sally’s belief.

The key move is holding a proposition the system knows to be false.

A naive agent has one state:

ball = box

An agent with a perspective model has at least two:

world.ball = box
sally.belief.ball = basket

Answer “Where is the ball?” from world; answer “Where will Sally look?” from sally.belief.

Sally’s belief is not a random error. It is stale but internally sensible because she did not see Anne move the ball.

The Sally-Anne task is a useful probe, not proof of robust ToM. A system might have memorized the story pattern or use shallow cues without maintaining a reusable model of Sally.

What an agent must model

To build ToM into a system, track at least four things:

• Belief: what the other agent treats as true.
• Observation history: what it has actually seen, including when it saw it.
• Goal: the outcome it is trying to produce.
• Capability and policy: what it can do and how it chooses actions.

In the incident, the monitoring agent saw the alert at 03:03 and the signed rollback event at 03:04. It did not observe a post-rollback error-rate measurement. Its view of service health is therefore stale.

A memory system might store “the rollback happened.” ToM asks, “Does the monitoring agent know that the rollback happened, and does it have evidence that the error rate recovered?” The event and ownership of that information are separate facts.

A compact model could look like this:

monitoring_agent:
  observed_until: 03:04
  last_rate_measurement: 03:03
  believes:
    version: v41
    errors_per_minute: 120
  goal: protect customers
  policy: page if errors_per_minute reaches 50
  confidence: high

Meanwhile:

world:
  version: v41
  errors_per_minute: 3
  rollback_completed: 03:04

There is no contradiction. The contradiction appears only when a system assumes every agent automatically sees world.

The mechanism: from observations to action

The causal mechanism has four steps:

1. The world changes.
2. Agents receive different observations.
3. Each agent updates its belief state from those observations.
4. It chooses an action from its belief, goal, and policy.

In compact form:

action = policy(belief_state, goal, available_actions)

ToM adds a perspective model:

predicted_other_action =
  other_policy(estimated_other_belief, estimated_other_goal, other_capabilities)

The word “estimated” matters. An agent infers another agent’s state from messages, tool traces, prior behavior, and access permissions. It can be wrong.

At 03:04, the deployment agent knows the rollback succeeded and knows the monitoring agent’s last rate measurement was at 03:03. It should predict that the monitor will continue to page: the rollback event changes the version belief but does not prove that the error rate is below the threshold.

A useful message changes the other agent’s information state. At 03:09, the deployment agent might send:

Rollback to v41 completed at 03:04 UTC.
The error rate was 3 per minute, measured from 03:04 through 03:09 UTC.
Your 03:03 alert is stale. Please rerun your check.

The timestamp explains the mismatch, the measurement supplies evidence, and the request identifies the next action. Once the monitoring agent receives and trusts it, its belief changes and it can stop paging.

This is information asymmetry: agents inhabit the same world but possess different information. ToM matters most when observations are partial, delayed, or private. If every agent has an instantaneous, complete, trusted view, there is little perspective-taking to do.

A worked example with numbers

Suppose the monitoring agent uses this rule:

• page when the estimated error rate is at least 50 per minute;
• close the incident below 50.

At 03:03 it observes 120 errors per minute. At 03:04 the service is rolled back, and the monitoring agent receives the signed rollback event. At 03:09 a fresh measurement covering 03:04 through 03:09 reports 3 per minute.

Before that fresh measurement arrives:

true state:       version v41, 3 errors per minute
monitoring view:  version v41, 120 errors per minute

The monitoring agent knows the version changed, but its latest rate evidence is still 120. Telling a human “the current error rate is 3” may be factually correct and operationally useless. The monitor will still page until its own view is updated.

world = {"version": "v41", "errors_per_minute": 3}

ravi_belief = {
    "version": "v42",
    "errors_per_minute": 120,
    "goal": "protect customers",
}

def choose_action(belief):
    if belief["errors_per_minute"] >= 50:
        return "page on-call"
    return "close incident"

print("Reality:", world)
print("Ravi chooses:", choose_action(ravi_belief))

ravi_belief.update(world)
print("After evidence, Ravi chooses:", choose_action(ravi_belief))

It prints:

Reality: {'version': 'v41', 'errors_per_minute': 3}
Ravi chooses: page on-call
After evidence, Ravi chooses: close incident

The crucial operation is not the threshold comparison. It is the separate ravi_belief state that can differ from world.

Production code must not blindly copy the world into an agent’s belief. The update should happen because a message was delivered, a permitted tool was called, or a shared-state contract says the value is observable. Preserve provenance as well:

claim: errors_per_minute = 3
source: monitoring_api
observed_at: 03:09
confidence: 0.98
visible_to: deployment_agent, monitoring_agent

The exact format is flexible; the invariant is not: an agent must not be credited with knowledge merely because another component knows the fact.

Do LLMs have theory of mind?

As of August 2026, the careful answer is: language models show ToM-like behavior, but that does not establish a stable, general-purpose theory of mind.

Large models can answer many false-belief questions, and performance often improves with capability, prompting, and additional reasoning. That is useful evidence, not proof of an internal belief model.

Classic stories are widely published. A model may have encountered the wording or close variations during training. Data contamination means evaluation material may have influenced the model before testing.

A verbal answer can also be produced without persistent state. A model may correctly say “Sally will look in the basket” and then, two turns later, answer as if Sally saw the move. Rewording, unfamiliar names, multiple agents, or a tool call that changes the world can expose this weakness.

A model may also identify the statistically likely answer without representing beliefs in a reusable way. This is the difference between behavioral competence and mechanistic explanation. The behavior can be useful while the internal explanation remains uncertain.

For high-impact actions, enforce epistemic boundaries outside the model. If a reviewer has not approved a production change, the system should block execution even if the LLM claims the reviewer knows about it.

A production pattern

Treat perspective as data:

1. Keep a canonical world state containing deployment events, database records, tool results, and verified facts. It answers “what happened?” but not “who knows?”
2. Keep per-agent observations and derived beliefs separate. Represent unknown explicitly; rollback_status = unknown is safer than a confident guess.
3. Infer actions from the modeled belief, goal, and capabilities. Make the predicted mental state inspectable.
4. Treat messages as state transitions. Record sender, recipient, delivery status, evidence, and acknowledgement. Transmission is not knowledge until the recipient processes the message.
5. Test perspective-taking directly.

Useful tests include different event views, stale but internally consistent beliefs, delayed or dropped messages, world changes after planning, different agent goals, and reworded scenarios with unfamiliar names.

Ask two questions separately: “What is true?” and “What will agent B do given what agent B has seen?” Many systems answer the first while pretending they answered the second. Agent observability and agent evaluation provide practices for tracing these differences.

Limits and failure modes

Explicit belief states consume storage, prompt space, inference time, and engineering effort. The model of another agent can be wrong even when the world state is right. For a deterministic pipeline in which every step reads the same synchronized database, explicit ToM may add ceremony without improving the result.

Use it when agents have partial observations, different goals, delayed communication, or strategic interaction. Skip it when there is no meaningful other perspective to predict. A calculator does not need ToM; agents deciding who may deploy v41 probably do.

ToM is also capability-neutral. Better opponent modeling can improve coordination, but it can improve manipulation. Pair it with least privilege, approval gates, policy checks, and audit logs. A hypothesis about another agent’s goal should remain uncertain and evidence-based, not become a fact from confident language.

Quick check

Next

Theory of mind powers the believable interactions of generative agents and the opponent-modeling used in multi-agent debate. It also explains why shared transcripts alone do not create coordination. Continue with swarm intelligence systems, then return to building with LangChain.