Multi-agent reinforcement learning

At 3 a.m., two warehouse robots approach the same narrow aisle. If both turn left, they pass safely. If both turn right, they also pass safely. If one turns left while the other turns right, both get stuck, the order misses its dispatch window, and somebody starts looking for the incident dashboard.

Now imagine that both robots are learning. Robot A updates its policy after every episode. Robot B does the same. A’s best decision depends on what B currently does, but B’s behaviour keeps changing while A is trying to learn. The target moves under the learner’s feet.

That is the problem multi-agent reinforcement learning, or MARL, exists to study: several agents learning from interaction in the same environment. Single-agent reinforcement learning assumes one learner and an environment whose rules do not change merely because the learner is improving. MARL removes that comfortable assumption.

The important distinction is not “many agents exist.” A scripted fleet of robots is not automatically MARL. It becomes MARL when the agents, or the policies controlling them, update from experience.

The mental model: a game, not a lone learner

Single-agent RL is usually described as a Markov decision process, or MDP. An agent observes a state, chooses an action, receives a reward, and reaches a next state:

state → action → reward, next state

With multiple learners, the environment is better described as a Markov game. The next state and reward depend on the joint action, the tuple of actions chosen by all agents:

P(s', r | s, a_A, a_B)

Robot A controls only a_A, and may not observe the full state s or B’s action. A might see its own lidar and battery level but not B’s intended turn. This is partial observability: each agent sees only part of the information determining the outcome.

The global state includes everything relevant, such as both robots’ positions, velocities, loads, and routes. A local observation is what one robot can access at execution time. Training may use the global state from a simulator; a deployed robot cannot use information it was never given.

Why the target moves

If B’s policy stays fixed, A can estimate the consequences of its actions against stable behaviour. If B learns too, A instead faces a distribution such as:

P(s', r | o_A, a_A; π_B)

where o_A is A’s local observation and π_B is B’s policy. The distribution averages over B’s possible actions and hidden states. It may also depend on action and observation histories because a local observation is not always Markov-sufficient.

When B changes its policy, the distribution A is estimating changes. Old experience may describe a different world from the one A faces now. The warehouse rules did not change; the other learner did.

Here is the smallest useful demonstration. Both agents want to coordinate, but B alternates its policy after every episode:

payoff = {
    ("left", "left"): 1,
    ("left", "right"): 0,
    ("right", "left"): 0,
    ("right", "right"): 1,
}
b_policy = "left"

for episode in range(1, 5):
    a_best = max(
        ["left", "right"],
        key=lambda action: payoff[(action, b_policy)],
    )
    print(
        f"episode {episode}: B plays {b_policy:<5} "
        f"-> A's best response = {a_best}"
    )
    b_policy = "right" if b_policy == "left" else "left"

episode 1: B plays left  -> A's best response = left
episode 2: B plays right -> A's best response = right
episode 3: B plays left  -> A's best response = left
episode 4: B plays right -> A's best response = right

A’s best response flips every episode. This is non-stationarity: the learning problem changes because other agents learn or adapt. It is the central MARL problem.

Independent learning can still work when agents barely interact, an opponent is fixed, or the problem is small. But its replay data becomes suspect: a transition collected against an old teammate policy may no longer support the current update.

Reward is not the same as credit

MARL settings are usually:

• Cooperative: agents share an objective and often a team reward.
• Competitive: agents have opposing objectives, sometimes zero-sum.
• Mixed: agents cooperate in some ways while competing in others.

Cooperation creates credit assignment: determining which agents’ actions caused a team outcome.

Return to the two-robot game:

If both turn left, the team receives 1, but that reward does not say whether A chose well, B chose well, both did, or the team got lucky.

A centralized learner can compare counterfactuals:

• Q(state, A left, B left) = 1
• Q(state, A right, B left) = 0

Holding B fixed, A’s contribution is:

1 - 0 = 1

A difference reward uses this kind of counterfactual as an individual signal: how much better was the team because agent i was present? It can improve learning, but the counterfactual may require another simulation, a learned model, or an approximation. In tasks where several agents are jointly necessary, no single agent has a clean marginal contribution.

Centralized Training, Decentralized Execution

Centralized Training, Decentralized Execution, or CTDE, separates training from deployment:

• During training, a centralized critic—a learned estimator of expected future return—can see global information such as every agent’s observation, action, and sometimes the simulator’s full state.
• During execution, each decentralized actor—the policy choosing actions—uses only that agent’s available observation.

The critic evaluates; the actor acts. The critic is not secretly driving the robots at runtime.

CTDE helps in three connected ways. The critic can distinguish joint events such as “A left while B left” from “A left while B right,” reducing ambiguity caused by partial observations. It can use joint information to reduce variance, or noisy swings in updates. And it can estimate team value with the variables responsible for much of the apparent drift, rather than forcing each actor to infer the whole game alone.

Seeing the joint action does not automatically identify individual causation. For that, the critic can compare the observed event with counterfactual alternatives and produce an advantage: how much better an action was than the critic’s baseline. The actors then learn from this richer signal while remaining local at execution.

CTDE reduces non-stationarity; it does not eliminate it. Opponents can adapt, critics can be wrong, and partial observability may require memory through a recurrent policy or observation history.

Three algorithm shapes

QMIX suits cooperative, discrete-action tasks. It learns an individual utility for each agent and combines them through a monotonic mixing network. Because increasing an individual utility cannot decrease the team value, agents can select actions locally without enumerating every joint action. The same restriction means QMIX cannot represent every coordination relationship.

MADDPG gives each actor a centralized critic that can use joint observations and actions during training. It is aimed at continuous actions such as steering or throttle, where an action’s consequence depends heavily on what teammates did simultaneously.

MAPPO applies PPO-style clipped policy updates with decentralized policies and commonly a centralized value function. It is often a strong baseline, but rollout freshness, reward scale, episode handling, and update count still matter.

Independent Q-learning or PPO is the simplest baseline. It can be adequate when interaction is weak, but rapidly changing teammates make replay stale and can destabilize updates. Start simple when the interaction is simple; do not confuse a familiar algorithm with a solution to a moving target.

Scaling, deployment, and when not to use MARL

The joint action space grows quickly. If ten agents each have five discrete actions, there are 5^10 = 9,765,625 joint combinations. Factorization avoids enumerating them, but only by imposing assumptions. Parameter sharing reduces the number of policies when agents have similar bodies and roles; hierarchy can separate route planning from local movement, but introduces additional delays and credit ambiguity.

CTDE also has a data cost: training needs synchronized joint trajectories and often global simulator or telemetry data. A central critic may conflict with privacy or safety requirements. Decentralized execution does not mean decentralized data governance.

Evaluate actors with exactly their deployment observations, and log per-agent returns as well as team return. A high team score can hide free-riding, invalid actions, or one agent doing all the work. In competitive tasks, test against held-out opponents; in cooperative tasks, vary teammates and starting states.

MARL is appropriate when repeated interaction and adaptation are central: traffic signals responding to one another, robots coordinating under uncertainty, or competing policies learning against one another. If a fixed planner or a workflow agent solves the task reliably, adding several learners buys you a research project and a new pager.

A system with several LLM agents exchanging prompts is not automatically MARL. It becomes relevant when policies repeatedly update from rewards such as delivery success, latency, cost, or user outcomes. Until then, workflow design, tool contracts, and evaluation are usually the more direct tools. See multi-agent orchestration for that distinction.

MARL has powered systems such as AlphaStar for StarCraft II and OpenAI Five for Dota 2, as well as research on traffic control, fleet coordination, and multi-robot control. The shared difficulty is:

Learning is harder when the environment is itself learning.

In one breath

• MARL is reinforcement learning with multiple agents learning in the same environment.
• Each agent changes the behaviour observed by the others, creating non-stationarity and stale experience.
• Team rewards create credit-assignment ambiguity because they do not identify individual contributions.
• CTDE gives the training critic a global view while keeping execution policies local.
• QMIX factorizes cooperative discrete-action values, MADDPG uses centralized joint-action critics, and MAPPO combines decentralized policies with centralized value estimation.
• Ask first: what can the actor observe at runtime, and which agent actually caused the reward?

Quick check

Next

MARL addresses agents that learn through repeated interaction. For the non-learning coordination layer, see multi-agent orchestration. For adversarial agreement among agents, see consensus and Byzantine fault tolerance. For measuring whether a learned team actually works, see agent benchmarks.