Decision-making under uncertainty

At 3:03 a.m., an account-security agent sees a login from a new country. The password is correct. The device is unfamiliar. The agent has three choices:

• lock the account;
• allow the login;
• spend two dollars and 400 milliseconds calling an identity-verification service.

A mistaken lock costs a customer a support call and ten minutes of lost access. A mistaken allow may cost the customer 100 dollars. The verification service is useful, but not free. Calling it ten times while the customer waits is not “being careful”. It is a different failure.

The agent must answer two questions:

1. Which action has the best likely consequence?
2. Is learning more worth the time, money, and risk of learning it?

“Pick the most likely answer” answers neither question. The framework for answering them is decision theory: choosing actions by combining beliefs about the world with the value and cost of their possible outcomes.

Expected utility, not just confidence

A state is the hidden situation that determines what happens; an action is what the agent does. Here the states are fraud and legitimate, and the actions are block and allow. A utility is a numerical score for each action-state pair. We measure it in dollars in this example:

Action	Fraud	Legitimate
Block	+$100	-$20
Allow	-$100	$0

The agent believes there is a 15 percent chance of fraud. Expected utility is the probability-weighted average of an action’s possible utilities:

EU(action) = sum of P(state) x U(action, state)

So:

EU(block) = 0.15 x 100 + 0.85 x (-20) = -$2

EU(allow) = 0.15 x (-100) + 0.85 x 0 = -$15

Blocking is better, even though it is less likely to be “correct”. Allowing is correct 85 percent of the time, but its 15 percent of expensive mistakes outweighs the cheaper mistakes from blocking. Expected utility prices both kinds of error.

Probability of correctness asks which state is most likely. Expected utility asks which action has the best average consequence. They diverge whenever mistakes have unequal costs.

The payoff matrix also gives an action threshold. If p is the probability of fraud:

EU(block) - EU(allow) = 220p - 20

Blocking is better when 220p > 20, or p > 1/11 ≈ 9.1%. The threshold is low because blocking gains $200 relative to allowing in the fraud state, while it costs only $20 in the legitimate state. Change the costs and the threshold changes too.

Expected utility does not replace hard rules. “Never transfer money without approval” is a constraint, not a preference to average with a token budget. Decision theory chooses among permitted actions.

Should the agent call another tool?

The value of information (VOI) is the improvement in expected utility from receiving an observation before acting.

Suppose the identity service has 90 percent sensitivity and a 10 percent false-positive rate. It costs $2. With a 15 percent prior fraud probability:

• P(positive) = 0.15 x 0.90 + 0.85 x 0.10 = 0.22
• P(fraud | positive) = 0.15 x 0.90 / 0.22 = 0.614
• P(negative) = 0.78
• P(fraud | negative) = 0.15 x 0.10 / 0.78 = 0.019

A positive result raises the fraud belief to 61.4 percent, making block worth about $53.64. A negative result lowers it to 1.9 percent, making allow worth about -$1.92; blocking would be about -$17.69. The results fall on opposite sides of the 9.1 percent action threshold, so the tool can change the decision.

The expected utility after observing the result, before paying for the call, is:

0.22 x 53.64 + 0.78 x (-1.92) = $10.30

The best action without the tool was blocking at -$2. Therefore:

• gross VOI: $10.30 - (-$2) = $12.30
• net incremental VOI after the $2 fee: $12.30 - $2 = $10.30
• expected utility after the call, including its fee: $10.30 - $2 = $8.30

The call improves the expected result by $10.30 relative to blocking immediately.

The stopping rule compares quantities in the same utility units:

call the tool when expected improvement > all-in cost

This calculation prices only the $2 fee. In production, convert latency, tokens, privacy exposure, and side effects into utility-equivalent costs, or enforce them as separate constraints and budgets. Raw milliseconds and dollars cannot be added directly.

If every possible result would lead to the same action, the information has no decision value. A tool can be accurate and still not be worth calling.

Explore or exploit?

Agents often choose repeatedly among tools, models, search strategies, or specialist agents whose quality is only partly known. A multi-armed bandit models this situation: each chosen option reveals a reward, while unchosen options reveal nothing.

Exploitation uses the option currently believed best. Exploration tries an uncertain option to learn whether it is better. Always exploiting can trap a system with its first adequate tool; always exploring wastes money. Bandits fit repeated choices with feedback, not one-off decisions with a known payoff matrix.

Thompson sampling maintains a probability distribution over each arm’s unknown quality. For binary success, an arm can use a Beta distribution representing successes and failures. Each round samples one success probability from every arm and chooses the highest; after observing the result, it updates only that arm. Sparse data leave a distribution wide, encouraging exploration; reliable data concentrate it around the arm’s measured quality.

This works best when rewards are reasonably stationary and arrive promptly. Vendor changes, delayed outcomes, context differences, and safety constraints require recency weighting, context, or hard exclusions. Do not route a payment approval to an untested specialist merely because its posterior is wide. Explore in a sandbox, on low-impact cases, or behind human approval.

The honest formalism: POMDPs

The login agent never observes “fraud” directly. It sees noisy clues such as location, device history, password status, and verification results. The hidden state can also change while it acts.

A partially observable Markov decision process, or POMDP, represents:

• States: hidden situations, such as legitimate user or attacker.
• Actions: operations such as challenge, block, or call a tool.
• Observations: evidence returned by the world or a tool.
• Transitions: how actions and time change the state.
• Rewards: the value of resulting outcomes.
• Discounting: how delay reduces the value of a reward.

“Markov” means the next state depends on the current state and action, not the entire history. Partial observability means the current state is hidden.

The agent maintains a belief state, a probability distribution over hidden states. Initially:

belief = {legitimate: 0.85, fraud: 0.15}

After a positive verification result:

belief = {legitimate: 0.386, fraud: 0.614}

Once updated correctly, the belief contains the information needed for the next decision; the full observation history is not required.

Exact POMDP solutions are usually impractical for language agents. Action and observation histories branch rapidly, beliefs are continuous, and world models are inaccurate. A policy that is optimal under false transition probabilities is still wrong.

The practical compromise is a belief-state heuristic:

1. Track only uncertainties that affect the next action.
2. Use a short lookahead.
3. Estimate which observation has positive net VOI.
4. Call one tool, update the belief, and reconsider.
5. Stop at an action threshold, escalation rule, or budget cap.

This is the reason behind many ReAct-style loops: decision theory determines whether another observation earns its cost. Planning patterns provides useful control structures.

Risk changes the threshold

Expected monetary value assumes linear utility: gaining a dollar offsets losing a dollar. Organisations may instead be risk-averse, making large losses hurt disproportionately. Represent this with a utility function, a tail-loss penalty, or a rule such as “the probability of a severe incident must remain below one in a thousand”. Otherwise an agent may optimise average reward while stakeholders care about the worst outcomes.

Irreversible actions deserve a higher evidence threshold than reversible ones. A wrong read-only search can be corrected; deleting production data, charging a card, sending a legal notice, or granting access may not be. The threshold rises because recovery is uncertain and the downside tail is costly.

A policy can allow many cheap observations automatically while requiring human approval for one irreversible write. The different thresholds reflect different consequences.

How the theory becomes engineering

Production systems turn the theory into small controls:

• Confidence thresholds should come from the payoff matrix and calibrated probabilities, not a familiar number such as 0.8.
• Escalation policies handle uncertainty, irreversible actions, or close utility comparisons. Staff time and delay are costs, not free oracle access.
• Budget caps limit tool calls, tokens, wall-clock time, and money. VOI decides how to spend the available budget.
• Belief logs record probabilities, evidence, chosen action, estimated utility, and outcome. This separates bad evidence from bad payoffs or thresholds.
• Approval gates separate reversible investigation from irreversible side effects. A planner can prepare a migration while policy requires approval before execution.

A practical controller is:

1. Estimate relevant state probabilities.
2. Reject actions forbidden by policy.
3. Choose the permitted action with highest expected utility.
4. Estimate each tool’s improvement minus its all-in cost.
5. Call the best worthwhile tool within the budget.
6. Update the belief and repeat, or escalate when no safe action meets its threshold.

This is decision theory translated into thresholds, routers, approval checks, and counters. Agent cost control covers the operational side of budgets; reflection covers related critique loops.

Failure modes

Symptom	Likely cause	Fix
High accuracy but costly fraud still slips through	The system optimises correctness, not asymmetric loss	Use a payoff matrix and expected utility
Several searches happen after the answer is actionable	There is no VOI stopping rule	Stop when another result cannot repay its cost
A noisy result triggers a destructive action	Overconfident belief update or low action threshold	Calibrate, corroborate, and raise the irreversible-action threshold
Costs rise without better outcomes	Information costs are missing from the model	Price latency, tokens, tools, recovery, and human labour

The limitation is unavoidable: decision theory needs probabilities and utilities, and agents are often bad at both. LLM confidence is not automatically calibrated; tool accuracy varies by user and time; stakeholders disagree about speed, privacy, and convenience. A belief-state heuristic makes assumptions visible but cannot make false assumptions true. Calibrate against held-out outcomes, review payoffs with the people who bear consequences, log decisions, and re-estimate after incidents. Use hard safety constraints where average-case optimisation is unacceptable.

What to remember

• Choose the highest expected utility, not necessarily the most likely correct answer.
• Information is valuable only when it can change the action enough to repay its cost.
• Explore unfamiliar tools safely; exploit known-good tools when feedback supports them.
• POMDPs model hidden state honestly; compact belief-state loops are the practical compromise.
• Irreversible actions need stronger evidence, approval rules, and larger downside penalties.

Quick check

0/3

Q1

Because probabilities are never useful in agent systemsBecause the costs of different mistakes can make its expected utility higherBecause the agent should always prefer the slower actionBecause confidence scores measure utility directly

Check answer

Q2

Yes, because the post-tool utility is higherYes, because any positive information value justifies a callNo, because the gross value is $2, less than the $5 tool costNo, because tools should only be called when confidence is below 50 percent

Check answer

Q3

Wait, because it is correct 80 percent of the timeWait, because rollback has any chance of inconvenienceRollback, because its expected utility is $8 versus negative $12 for waitingThey are equal because the probabilities add to 100 percent

Check answer