A model repeatedly emits malformed arguments and occasionally calls the wrong tool. What would you include in a tool contract and execution layer to make tool use reliable without relying on the model to behave perfectly?

How to think about it

The answer

I would treat every model tool call as an untrusted request: define each tool with a strict machine-readable input and output contract, then put a deterministic execution layer in front of the real service. That layer parses, validates, authorizes, checks business rules, handles bounded retries and idempotency, and refuses unsafe calls instead of assuming the model will eventually behave.

Why this works

An LLM is not a typed client. It predicts a sequence of tokens that resembles a tool call. Sometimes those tokens contain valid JSON. Sometimes they contain a number as a string, an extra field, a missing required field, or a perfectly valid argument for the wrong operation.

A schema addresses the first class of problem. It defines the exact shape and types of acceptable arguments: required fields, allowed values, minimums, maximums, patterns, and whether unknown fields are rejected. It should also define the result shape and structured error codes. A human-readable description still matters because the model uses it to choose among tools, but the description is not enforcement.

The executor must make a separate decision. It should ask:

• Is this tool name in the application’s allowlist?
• Is this caller allowed to use it for this tenant, user, and resource?
• Do the arguments pass structural validation?
• Do they pass business and safety rules?
• Is this operation read-only or does it create an external side effect?
• Can it be retried safely?
• Does the request need user confirmation?

That separation is the mechanism the interviewer is probing for. The model proposes an operation. The application decides whether that operation is legal.

A strict schema can tell you that amount_cents is an integer. It cannot tell you that the customer owns the order, that the order has not already been refunded, or that the user actually asked for a refund rather than a refund-status lookup. Those checks belong in the execution layer.

A concrete contract

Suppose we are building a support agent for an online shop. It has two tools:

Tool	Purpose	Side effect
get_refund_status	Read whether an order already has a refund	None
create_refund	Create a refund for an order	Moves money

Their descriptions must be deliberately non-overlapping. get_refund_status should say that it never creates or changes a refund. create_refund should say that it creates a financial side effect and requires an explicit confirmation state supplied by the application.

The input schema for create_refund might look like this:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "order_id",
    "amount_cents",
    "currency",
    "reason"
  ],
  "properties": {
    "order_id": {
      "type": "string",
      "pattern": "^ord_[A-Za-z0-9]+$"
    },
    "amount_cents": {
      "type": "integer",
      "minimum": 1,
      "maximum": 100000
    },
    "currency": {
      "type": "string",
      "enum": ["USD"]
    },
    "reason": {
      "type": "string",
      "enum": [
        "duplicate",
        "customer_request",
        "damaged"
      ]
    }
  }
}

For a twenty-five-dollar refund, the model must produce 2500 cents, not 25.00 dollars and not the string "2500". Integer minor units avoid floating-point currency errors. Rejecting additional properties is useful too: an unexpected field often signals that the model has mixed two tools together.

Now imagine the user says, “Refund order ord_4821 for $25 because it was charged twice.” The model emits an argument object with "amount_cents": "2500" and "status": "pending". The executor rejects it before making a network request. It returns a compact, structured error such as invalid_arguments, identifies the two offending fields, and allows one repair attempt.

After repair, the shape may be valid. The executor still fetches the order and checks the facts. Suppose ord_4821 is a $40 order with $20 already refunded. The remaining refundable amount is $20. The schema accepts 2500, but the semantic check rejects it because the requested amount exceeds the refundable balance. The application must not quietly reduce the refund to $20. It should ask the user to confirm a corrected amount.

This is also how the wrong-tool case should fail safely. If the user asks, “Has my refund arrived?” and the model selects create_refund, a schema may accept the arguments perfectly. The executor’s write policy should still block it because there is no confirmation for a new financial side effect. The application can ask a clarification question or route the user to the read-only status operation. A valid shape is not evidence of valid intent.

What the execution layer should do

A practical execution path is:

1. Parse only the expected format. Do not extract JSON from arbitrary prose and pretend the call was clean.
2. Look up the exact tool name in a server-side registry. Unknown names are rejected; the model cannot supply an arbitrary URL, SQL statement, shell command, or implementation.
3. Validate the arguments against the input schema. Do not silently coerce "2500" into 2500 unless that conversion is an explicit, tested part of the contract.
4. Run semantic checks such as resource ownership, tenant boundaries, current account state, amount limits, and confirmation requirements.
5. Attach server-controlled identity, authorization, and correlation data. Never trust the model to provide the user ID or permission scope.
6. Execute with a deadline, constrained credentials, and a bounded response size.
7. Validate and sanitize the result against an output schema before returning it to the model.
8. Record the tool name, schema-validation result, policy decision, latency, error class, retry count, and a redacted argument summary.

Malformed arguments deserve a bounded repair loop, not an infinite conversation with a stubborn autocomplete engine. I would normally allow one repair turn. A structural error such as a missing field can be shown to the model in machine-readable form. A policy error such as “refund exceeds remaining refundable amount” should usually go back to the user, because the model must not invent a new amount.

Retries need different rules. An invalid argument or authorization failure is not transient and should not be retried. A 429 rate-limit response or a temporary 503 may be retried with exponential backoff and jitter, perhaps twice, but only when the operation is idempotent. An idempotency key lets repeated attempts represent one logical operation rather than two refunds. The key must be controlled by the application and stable across retries.

The senior nuance

“Use JSON Schema” is necessary but not sufficient. Schema validation solves malformed structure; it does not solve tool selection, authorization, prompt injection, stale state, or duplicate side effects.

The opposite mistake is making the contract so rigid that normal user requests cannot be represented. Do not force the model to guess an enum value when the user’s reason is genuinely ambiguous. Separate extraction from commitment: collect the proposed action, show the exact amount and target, then obtain application-controlled confirmation before executing a consequential write.

Also, a timeout does not prove that the operation failed. The server may have accepted a refund and the response may have disappeared. Retrying immediately can duplicate the side effect. For an unknown outcome, first query using the idempotency key or operation ID. Reliability is not merely “the call eventually succeeds”; it is “the system reaches one known state.”

One common production symptom is a growing stream of 400 invalid_arguments errors followed by occasional duplicate writes after someone adds a blind retry. The fix is not a larger retry count. It is strict classification: repair or clarify invalid input, retry only known transient failures, and make writes idempotent.

What they’ll ask next

How do you detect that the model chose the wrong tool if the arguments are valid?

Schema validation cannot do that reliably. Use sharply separated tool descriptions, expose only the relevant tools where possible, enforce read-versus-write policy in the executor, and require confirmation for high-impact operations. When intent remains ambiguous, ask rather than guess.

Would you let the model repair its own malformed call?

Yes, but with a small budget and no side effect before validation. Return a structured error, allow one corrected call, then stop and ask the user or fail clearly. Never let repair become an unbounded loop.

How would you retry a tool that charges a customer?

Only with an application-generated idempotency key and a server that honors it. Retry known transient failures with a bounded backoff. If the outcome is unknown, query the operation state before attempting anything again.

One line to say in the room

“Treat the model as an untrusted planner: schemas catch malformed calls, but a deterministic, authorized, idempotent executor decides whether any proposed tool call is allowed to happen.”