What is an AI agent, and how does it differ from a single LLM call?

How to think about it

An AI agent is an application that puts a large language model, or LLM, inside a bounded loop where it chooses actions, receives results, and decides what to do next. A single LLM call produces one response and stops; any tool access, memory, or side effect comes from the surrounding application, not from that call alone.

Why interviewers ask this

An LLM maps a prompt and its supplied context to an output. That output might be prose, JSON, or a request to call a tool. A tool is an external capability such as a database lookup, web search, payment API, calculator, or code runner.

The model does not automatically have access to those systems. It cannot refund an order merely by writing, “The refund has been processed.” It has generated text. The application must decide whether that text represents an allowed action, execute the action, and provide the result back to the model.

An agent adds a control loop around the model:

1. The application gives the model a goal and the current state.
2. The model returns either a final answer or a proposed action.
3. The runtime, meaning the program that runs the agent, validates and executes the action.
4. The tool returns an observation, meaning the result of that action.
5. The application adds that observation to the state and calls the model again.

The loop stops when the model produces a final answer, reaches a step or time limit, encounters an error, or needs human approval. This is bounded autonomy: the model may choose the next step, but only among the capabilities and policies the application exposes.

When people say an agent “reasons,” the useful engineering meaning is that the model selects the next action based on the current state. It does not mean that a reliable, inspectable chain of thought is necessarily exposed.

A concrete example

Suppose a customer writes at 10:02:

“I was charged twice for order 8472. Please refund the duplicate $249 charge.”

A single LLM call can draft a polite reply. If the order details are not in the prompt, however, it cannot know whether the charge exists. It cannot inspect the payment system. If the application looks up the order first and includes the result in the prompt, the LLM can summarize that result, but the lookup was still performed by application code outside the single call.

An agent could handle the task like this:

• The first model call chooses lookup_order with order ID 8472.
• The tool returns two captures of $249, both made at 10:01, and says the order was delivered.
• The next model call chooses check_refund_policy.
• The policy tool returns that duplicate captures are refundable within 30 days.
• The model proposes issue_refund for the second transaction.
• The runtime checks the amount, transaction ID, user permissions, and whether approval is required.
• After approval, the payment tool returns a refund ID.
• The model writes a final response containing that refund ID.

The model is choosing the next step, but it is not directly moving money. The runtime remains responsible for authorization and execution.

A simplified version looks like this:

# Illustrative pseudocode, not a vendor SDK.
state = [user_message]

for step in range(6):
    decision = llm(state, tools=tool_schemas)

    if decision.kind == "final":
        return decision.text

    call = validate_and_authorize(decision)
    result = execute(call)
    state.extend([decision, {"role": "tool", "content": result}])

raise RuntimeError("step limit reached")

The important line is the one that adds the tool result to state. Without that observation, the next model call would be guessing about the world rather than responding to what actually happened.

There is a real cost to the loop. If five model invocations each take 600 milliseconds and the three tools each take 150 milliseconds, the serial path takes about 3.45 seconds before queueing or human approval. It also uses roughly five times the per-call inference opportunity, although the exact token cost depends on how much state is sent each time. More context is often resent on every turn unless the serving system manages conversation state separately.

The practical distinction

Property	Single LLM call	Agent
Control flow	One application-defined pass	A loop can continue based on model decisions
External data	Must already be in the prompt	Can fetch data through tools
Side effects	None by itself	Possible through validated tools
State	The supplied prompt and context	State can include tool results and task progress
Cost and latency	Usually one model invocation	Multiple model and tool operations
Failure surface	Mostly output quality	Output, tools, permissions, loops, timeouts, and stale state

Common mistake: a model returning a tool call does not mean the tool has run. A single invocation may produce lookup_order(...) as structured output. The host must execute it. If the host executes exactly one tool call and then stops, that is tool calling, not a multi-step agent in the useful sense.

The senior-level nuance

An agent is an architecture, not a special kind of model. The same model can be used as a one-shot classifier, a fixed workflow step, or the decision-maker inside an agent loop.

The word “agent” is also used loosely. A workflow with four hard-coded steps and one LLM call for extracting an address is usually better described as a workflow. A system that lets the model decide whether it needs an order lookup, a fraud check, or a shipping search is more agent-like. The boundary is not mathematical, so it helps to state the operational definition being used.

Memory is another source of confusion. Short-term state is the messages and tool results carried through the current task. Long-term memory means data saved outside the conversation, such as customer preferences in a database, and retrieved later. Neither appears automatically because a model is placed in a loop. The application has to store, retrieve, filter, and sometimes delete it.

For a known process, deterministic code is often safer than an agent. If every refund must follow the same three checks, encode those checks directly and use the LLM only to understand the customer’s request. Use an agent when the number or order of useful steps is genuinely variable, the task needs live systems, and the value of flexible decisions outweighs extra latency and failure modes.

A production agent should therefore have an allowlist of tools, strict argument validation, least-privilege credentials, a maximum step count, a deadline, and a token or cost budget. Irreversible actions such as refunds should require approval when appropriate and use an idempotency key, so a retry cannot issue the same refund twice. Logs should record the task, model version, tool calls, results, latencies, and final outcome.

A failure mode to recognize

At 3 a.m., the first symptom may be a rising queue of requests, repeated lookup_order calls in the trace, and payment-service errors such as 429 Too Many Requests. The cause is often a missing stop condition: the tool returned an empty result, the model interpreted that as a reason to retry, and the runtime allowed the loop to continue.

A step limit and deadline stop the runaway task. Structured tool errors give the model a useful failure state. Duplicate-call detection and idempotency protect external systems. The fallback should say that the request needs human review, not quietly invent a successful refund.

What they’ll ask next

Can a single LLM call use tools?

Yes, it can return a structured tool-call request. But the host application still has to execute the tool. If there is no follow-up call that supplies the result and lets the model continue, it is a tool-enabled one-shot interaction rather than a full agent loop.

Is retrieval-augmented generation, or RAG, automatically an agent?

No. A RAG system can retrieve documents once and pass them to one LLM call. It becomes agent-like when the model decides what to search, whether another search is needed, which source to use, or what action to take after retrieval.

Do agents need multiple agents?

No. One LLM with several tools and a reliable runtime is enough. Multiple agents add communication and coordination overhead, so they are justified only when separate roles, permissions, or parallel work solve a real problem.

Say this in the interview

“An agent is an LLM inside a bounded tool-use loop: it proposes an action, the runtime executes and validates it, the result goes back into state, and the process repeats; a single LLM call produces one output and stops.”