You are choosing an agent runtime for a team that needs durable state, middleware, observability, streaming, and portability across model providers. How would you compare runtimes such as Agno and Mastra, and what proof-of-concept results would determine the choice?
I would run Agno and Mastra against the same failure-injected workflow and choose on measured durability, middleware behavior, trace completeness, streaming quality, provider portability, and operating cost. Mastra is a natural first candidate for a TypeScript team, while Agno is a strong first candidate for a Python team, but the proof of concept makes the decision.
How to think about it
I would run Agno and Mastra against the same failure-injected workflow and choose on measured recovery, trace completeness, streaming behavior, and provider-switch cost—not on feature count. Mastra is a natural first candidate for a TypeScript team already using the AI SDK ecosystem, while Agno is a strong first candidate for a Python team that values a compact, model-agnostic agent, team, and workflow surface; the proof of concept makes the decision.
Why the runtime matters
An LLM call is not durable. If the process dies after the model says “submit the claim” but before the tool call finishes, the model does not remember what happened. The runtime must record execution state somewhere reliable and decide what to retry, what to resume, and what must never happen twice.
That makes an agent runtime more than a convenient wrapper around a chat-completions API. It is the control plane around the model:
- State records the conversation, tool results, workflow position, approvals, and errors.
- Middleware intercepts requests and events for authentication, tenant limits, redaction, retries, budgets, and policy checks.
- Observability turns a run into a trace: model calls, tool calls, latency, token usage, errors, and state transitions connected by one correlation ID.
- Streaming carries incremental tokens and meaningful events such as “tool started”, “approval required”, and “workflow resumed”.
- Provider abstraction lets the application change model vendors without rewriting every agent and tool.
The important distinction is between memory and durability. Memory means the agent can use previous conversation turns. Durability means a worker can be killed, replaced, or restarted and continue from a known checkpoint without repeating an irreversible action.
Agno and Mastra both cover the agent-building surface, but they arrive with different gravitational pulls. Agno is Python-first and presents primitives for agents, teams, tools, workflows, storage, and model integrations. Mastra is TypeScript-first and is closely aligned with the AI SDK ecosystem, with agents, tools, workflows, storage, streaming, and observability facilities.
I would compare the actual execution semantics, not the labels in the documentation. “Has memory” is not enough. I want to know whether a suspended workflow survives a deployment, whether a retry can duplicate a payment, and whether a trace shows the policy decision that blocked the tool.
The comparison I would make
| Question | Agno | Mastra |
|---|---|---|
| Best initial fit | Python services, data tooling, and teams already comfortable with Python | TypeScript services and teams already using the AI SDK ecosystem |
| Agent composition | Agents, teams, tools, workflows, and model integrations | Agents, tools, workflows, model-provider integrations, and application services |
| State | Session storage and memory options; workflow durability depends on the configured runtime and storage | Workflow state, storage, and suspend or resume patterns; exact behavior depends on workflow and storage configuration |
| Middleware | Compare hooks around requests, model calls, tools, and workflow steps | Compare request, model, tool, and workflow extension points, including the surrounding AI SDK layer |
| Observability | Runtime and deployment tooling can expose execution monitoring; verify exporters and span detail | Built-in observability and tracing integrations are a strong part of the ecosystem; verify exporter and retention behavior |
| Streaming | Test token streaming and event streaming separately | Test token streaming, workflow events, and reconnect behavior separately |
| Portability | Broad provider abstraction, but provider-specific behavior still matters | AI SDK provider abstraction is useful, but it does not erase model capability differences |
That table identifies hypotheses. It does not prove them.
For example, a framework may stream tokens beautifully while hiding tool events. It may persist a session but not the exact point at which a workflow can safely resume. It may emit a dashboard trace while dropping the tool arguments that explain why a production decision went wrong.
A concrete proof of concept
I would use one scenario rather than five toy demos: a customer claims assistant.
The assistant receives a claim, looks up the customer’s policy, estimates coverage, asks for human approval when the amount exceeds 1,000 dollars, and submits the approved claim to a claims system. The final submission is an irreversible side effect.
The durable state should include the claim ID, policy version, extracted fields, tool results, approval status, workflow step, model decisions, and an idempotency key for submission. The runtime should checkpoint before the submit step. The claims API should also reject a repeated idempotency key. Durability is a system property, not a checkbox in the agent library.
I would implement the same tools and prompts in both runtimes and run 300 scripted cases:
- 200 normal claims
- 50 malformed or incomplete claims
- 50 cases with injected model, network, storage, and worker failures
I would first use the same model provider in both runtimes so the initial comparison is about orchestration rather than model quality. Then I would repeat the suite with a second provider. The application should change only its model configuration and any explicitly documented provider adapter, not its business logic.
The gates would be concrete:
| Test | Example acceptance gate |
|---|---|
| Worker killed during execution | All 100 injected crash points resume or fail visibly; zero duplicate submissions |
| Deployment restart | A run survives deletion of the worker’s local disk and resumes from external storage |
| Human approval | The workflow waits for approval for 24 hours without burning model calls or losing context |
| Streaming | First visible event arrives within 1.5 seconds at p95; the client receives token, tool, approval, and completion events |
| Observability | Every run has one trace containing every model and tool call, with customer secrets redacted |
| Provider switch | The same contract tests pass on both providers; tool-call validity, structured-output validity, latency, and cost are measured separately |
| Operations | The team can inspect a stuck run, replay a safe step, and identify its tenant and deployment from the trace |
Those numbers are example gates, not universal truths. A voice assistant may need a tighter first-event target. A back-office claims process may accept five seconds but demand perfect recovery. The product’s failure cost sets the threshold.
I would also record cost per successful claim, not cost per model call. A retry that makes a run 20 percent more expensive is acceptable if it prevents a duplicate payment. It may be unacceptable for a low-value support question.
The senior-level nuance
The textbook answer is often “pick the most portable framework.” Portability is useful, but it has two layers.
The first is syntactic portability: changing a model identifier without rewriting the application. Both ecosystems can help with that.
The second is semantic portability: preserving behavior when the provider changes. Models differ in tool-call formatting, structured-output reliability, context limits, streaming events, reasoning behavior, and refusal patterns. A runtime cannot normalize those differences away. Contract tests and provider-specific fallbacks still belong in the application.
Middleware has a similar trap. A retry around a read-only policy lookup is sensible. A blind retry around claim submission is dangerous. Redaction must happen before data reaches telemetry, not merely before it reaches a dashboard. Authentication and tenant authorization must happen before the model can select a tool. I would test middleware order explicitly, because a correctly implemented check in the wrong place is still a security bug.
The storage adapter matters as much as the runtime. A local SQLite file may be perfect for development and a poor choice for several workers across availability zones. I would test with the production database, deployment model, queue behavior, backup policy, and retention settings. If the runtime cannot provide the required guarantees, I would put a dedicated durable workflow engine underneath it and use Agno or Mastra for model-facing steps. An agent framework should not quietly become the ledger for money movement.
A likely failure mode is easy to spot: after a worker restart, the customer is asked for approval again, or the claims system receives two submissions. In traces, you may see two unrelated run IDs and no durable checkpoint between “submit requested” and “submit completed.” That symptom tells me the system has conversation persistence, not reliable workflow recovery.
What they’ll ask next
“Which one would you choose today?”
I would start with the team’s language and deployment constraints. For an existing TypeScript platform, I would prototype Mastra first. For a Python platform with substantial data and tool code, I would prototype Agno first. Neither gets the production decision without passing the same recovery and observability gates.
“How do you test observability?”
I would inject a known claim containing a fake API key and customer email, then verify that the trace contains the run, model call, tool call, latency, and policy decision while containing neither secret. I would also kill a run and confirm that the resumed trace remains connected to the original request.
“How do you avoid vendor lock-in?”
Keep business tools behind application-owned interfaces, store durable state in a system we control, use provider contract tests, and export traces through an open telemetry path where practical. Changing the runtime should still cost something; the goal is a controlled migration, not the fantasy of zero migration cost.
What I’d say in the room: “I would choose the runtime that survives a killed worker without duplicating the side effect, explains every step in a trace, streams the events our UI needs, and passes the same provider-switch contract tests—not the one with the longest feature list.”