Designing human-agent interaction

At 3:07 p.m., Maya asks her operations agent to check whether the team should renew a software subscription.

The agent reads twelve invoices, finds a renewal notice, compares this year’s usage, and replies:

Renewal looks safe. Approve the $480 charge?

Maya is in another meeting. She clicks Approve.

Ten minutes later, finance notices that the $480 is not the renewal. It is a cancellation fee from a different vendor. The agent had matched the wrong company name in a scanned PDF. Its answer sounded calm because language models are very good at sounding calm.

Maya now has two bad choices. Trust the agent again and risk another charge. Or stop using it and do the work herself. The product has trained her into over-trust first, then abandonment.

This is the trust calibration problem: the user’s confidence in the agent should roughly match the agent’s actual reliability in the current situation. An agent that hides uncertainty invites dangerous trust. One that interrupts constantly teaches the user to click through every prompt.

Human-agent interaction is therefore the control surface for a system that can inspect, decide, and change things.

Show the work that matters

A useful interface answers four questions:

1. What is the agent trying to accomplish?
2. What has it found or changed?
3. What does it want to do next?
4. What can I correct if it is wrong?

Do not show a transcript of internal thought. Hidden chain-of-thought is not a reliable audit trail or a good interface: it can contain guesses, discarded ideas, and persuasive nonsense.

Show a compact rationale backed by recorded tool outputs. Link observations to an exact source, page, or data span, and distinguish extracted facts from inferences. For Maya:

• Goal: decide whether to renew Acme Analytics.
• Facts: renewal-notice.pdf, page 1, contains OCR spans Acme Analytics and $480; the search found 12 invoices, 9 marked paid.
• Inference: the notice belongs to the Acme workspace because its invoice ID appears in the ledger result.
• Missing: the cancellation-terms search found no matching document.
• Next action: draft a message asking the vendor to confirm the renewal date.
• Approval: sending the message externally.

This gives Maya a verifiable lead. She can open the cited page and compare the invoice ID without reading a 2,000-word model monologue.

Progressive disclosure

Progressive disclosure shows a simple answer first and reveals detail when needed. It balances auditability with speed:

• Result: “Found a possible renewal for $480 due 30 September.”
• Decision context: the matching invoices, source document, calculation, and assumptions.
• Action record: the tool, time, resource, result, and failures for each event.

The first layer supports speed, the second correction, and the third investigation. A collapsed view should still be specific: “Checked 12 invoices in Finance/Subscriptions; 9 matched Acme Analytics; 1 renewal notice found.”

The stable mental model is: gather evidence, propose a change, pause when it matters, and record the result.

Ask versus act

The wrong question is “How confident is the model?” Confidence describes a prediction; approval concerns consequences.

Base interruption on:

• Reversibility: can the action be cleanly undone?
• Consequence: what does being wrong cost?

Consequence includes external communication, deleted data, privacy exposure, legal commitments, reputation, and time—not only money.

Situation	Default interaction	Why it fits
Read files, search records, calculate a draft	Act, then report	Private and repeatable
Create a draft or temporary branch	Act and show it	The user can inspect or discard it
Send an email or post publicly	Preview and ask once	External audience and wording matter
Change a setting	Ask with exact before and after	The change matters, even if reversible
Delete data, transfer money, or grant access	Explicit approval	Recovery is costly or uncertain
Repeat similar low-risk updates	Batch a bounded set	One review is faster when scope is visible

The unit is user-visible consequence, not “tool call.” One tool call could send fifty emails; fifty tool calls could create harmless drafts.

A worked approval policy

This simple policy approves actions that are irreversible, external, or above a dollar limit. It does not use model confidence.

from dataclasses import dataclass


@dataclass(frozen=True)
class Action:
    name: str
    reversible: bool
    cost_usd: float
    external: bool


def needs_approval(action: Action, cost_limit: float = 25.0) -> bool:
    return (
        not action.reversible
        or action.external
        or action.cost_usd >= cost_limit
    )


actions = [
    Action("read invoices", True, 0.00, False),
    Action("draft email", True, 0.00, False),
    Action("send email", False, 0.00, True),
    Action("renew subscription", False, 480.00, True),
]

for action in actions:
    decision = "approve" if needs_approval(action) else "act"
    print(f"{action.name}: {decision}")

It prints:

read invoices: act
draft email: act
send email: approve
renew subscription: approve

This is a legible starting point, not a complete safety policy. Production rules should also consider data sensitivity, recipient count, permission changes, rate limits, and whether an undo is real.

For Maya, the agent can read invoices, calculate totals, and prepare a recommendation automatically. It should ask before sending the vendor email or renewing the subscription.

Plan approval is not step approval

Asking after every action creates approval fatigue:

1. May I open the invoice?
2. May I compare the amount?
3. May I draft the email?
4. May I send the email?

After several routine prompts, users click the next one reflexively. Instead, present a bounded plan:

I will compare 12 invoices, identify the current renewal, draft a confirmation email, and show it before sending. I will not send messages or make purchases without another approval.

Approval covers the plan’s scope, not an unlimited licence. If the plan changes materially—such as finding a second vendor—the agent must stop and ask again.

Batch only items with the same action, permission, and risk. Show counts and exceptions:

Approve 18 read-only label changes?
16 match the rule. 2 conflict with existing labels. Review exceptions.

Do not combine an irreversible purchase with routine updates behind one Approve all button. Show exact scope, before-and-after state, affected recipient or system, approval expiration, and controls to edit or narrow the plan.

Be honest about uncertainty

Useful uncertainty identifies:

1. Observed: “The PDF contains $480 and the name Acme Analytics.”
2. Inferred: “I matched it by invoice ID in last month’s ledger.”
3. Missing: “Cancellation terms were not found.”

Compare:

Confidence: 87 percent. Renewal is safe.

with:

I found one renewal notice for $480. I matched it to Acme Analytics by invoice ID, but the scanned vendor name is partly unreadable and I did not find the cancellation terms. Verify the vendor before approving payment.

The second message is easier to correct because it exposes the uncertainty’s shape.

A numerical probability is useful only when tied to a tested, calibrated prediction; it does not establish that paying an invoice is a good decision.

Progress is part of the interaction

A blank screen for eight seconds feels broken. Useful milestones feel like work because silence leaves the user unsure whether the agent is running, stuck, or waiting.

For Maya:

• Started: reviewing Finance/Subscriptions.
• Found: 12 invoices; 9 matched Acme Analytics.
• Extracted: $480, due 30 September.
• Needs review: vendor name is unclear in one scan.
• Ready: recommendation and email draft prepared.

Do not stream fake “thinking” messages. Emit events when a source is found, a tool completes, a plan changes, or a human decision is required.

Say explicitly when waiting: “Waiting for your approval to send the email.”

Failure should leave a useful state

Permission errors, rate limits, malformed documents, and ambiguous tool results are inevitable. Graceful failure preserves completed work and gives the user a next step:

I reviewed 12 invoices and prepared the renewal comparison. I could not read the vendor name on invoice 12, so I did not send the email or renew the subscription.

You can: upload a clearer copy, edit the vendor name, retry extraction, or send the draft yourself.

Checkpoint meaningful stages: source selection, extracted facts, drafts, proposed plan, approved scope, and each external action.

Save the artifacts—not just “failed”: the draft, source references, amount, and unresolved question.

A handoff should include the original goal, completed and unattempted actions, current result, assumptions, errors, and the next human decision. The human should not have to interrogate the agent to recover its work.

Workflow systems such as human-in-the-loop with LangGraph provide durable pause and resume points. The interface still has to make that pause understandable.

Undo is a product feature, not a database feature

“Undo” depends on the side effect:

• A draft can be discarded.
• A file move can often be reversed.
• A sent email may already have been read.
• Revoking permission cannot erase exposed data.
• A refund compensates for a payment; it does not reverse it.

Do not call a compensating action Undo if the original effect cannot be erased. Use Send correction, Restore access, or Request refund.

A user-facing history should show who or what acted, when, the target and change, the approval or policy version, scope, result, and recovery action.

For Maya:

14:42 — Agent drafted renewal email.
14:43 — Maya approved sending to billing@acme.example.
14:43 — Email sent. Message ID recorded.
14:44 — Subscription renewal not attempted.

Failure modes you can see first

• Approval rate is nearly 100 percent: prompts are too frequent or too vague. Batch homogeneous low-risk work, separate exceptions, and measure correction and review time—not approval alone.
• Users stop execution after one mistake: the interface hid evidence and assumptions. Show both beside consequential claims and make the first execution narrow and reversible.
• People click twice or lose a handoff: stream real milestones, distinguish running, waiting, completed, and outcome unknown, and checkpoint work products.

Interaction design cannot compensate for excessive permissions or unsafe tools. See agent safety controls for enforcement and the coding-agent workbench for an applied example.

The honest limitation

No interface can make an unreliable agent reliable. Evidence can be incomplete, logs can faithfully record bad decisions, and undo cannot recover a secret copied elsewhere. More disclosure can also overwhelm busy users.

The goal is appropriate human attention: automate low-consequence work visibly, pause before consequential work, state what is known and assumed, and preserve control when the agent fails.

What to remember

• Calibrate trust with evidence, assumptions, and outcomes—not tone or confidence percentages.
• Ask based on reversibility and consequence, not model confidence.
• Use progressive disclosure, bounded batches, honest progress, checkpoints, and meaningful recovery.
• “Unknown” is safer than a fictional success.

Quick check

0/3

Q1

Send it automatically because the confidence is highAsk for approval and show the recipient, message, and consequenceAsk for approval after every read-only lookup firstShow a larger confidence percentage

Check answer

Q2

Confidence: 87 percentThe model thinks this is probably correctI matched the invoice by ID, but the scanned vendor name is unclear and cancellation terms were not foundNo uncertainty detected

Check answer

Q3

Put all three actions behind one Approve all buttonAsk for approval after every tool call, including each label updateAuto-apply the bounded label batch, show the announcement as a draft, and separately approve deletion and publicationLet the agent decide using its confidence score

Check answer