Multi-agent reinforcement learning

Single-agent reinforcement learning has one agent learning in a fixed environment. Multi-agent RL (MARL) has many agents learning simultaneously — and that simultaneity breaks the assumption single-agent RL is built on, creating problems that don’t exist with one learner.

Non-stationarity: the moving target

The central difficulty is non-stationarity. From any one agent’s point of view, the environment isn’t fixed — it changes as the other agents learn, so the “best action” is a moving target:

payoff = {("left","left"): 1, ("left","right"): 0, ("right","left"): 0, ("right","right"): 1}  # coordination game
b_policy = "left"

for episode in range(1, 5):
    a_best = max(["left", "right"], key=lambda a: payoff[(a, b_policy)])   # A best-responds to B
    print(f"episode {episode}: B plays {b_policy:<5} -> A's best response = {a_best}")
    b_policy = "right" if b_policy == "left" else "left"                   # B is also learning/changing

episode 1: B plays left  -> A's best response = left
episode 2: B plays right -> A's best response = right
episode 3: B plays left  -> A's best response = left
episode 4: B plays right -> A's best response = right

A’s optimal action flips every episode because B’s policy keeps changing. A single-agent RL algorithm assumes a stationary environment and would chase this moving target, often failing to converge. Non-stationarity is why you can’t just run independent single-agent learners and hope.

Settings, credit assignment, and CTDE

MARL comes in three flavors: cooperative (agents share a reward — a team), competitive (zero-sum — adversaries), and mixed. Cooperative settings add a second hard problem: credit assignment — when the team earns a reward, which agent’s actions deserve the credit? Without solving that, agents can’t tell good individual choices from lucky ones.

The dominant answer to both problems is Centralized Training, Decentralized Execution (CTDE): during training, use a centralized critic that sees all agents’ observations and actions — which makes the environment effectively stationary from the critic’s view and helps assign credit. At execution, each agent acts on only its local observation, so the system stays decentralized and scalable. The workhorses build on this: MADDPG (centralized critics per agent), QMIX (factor the team value into per-agent values for cooperative tasks), and MAPPO (multi-agent PPO).

In one breath

• MARL has many agents learning at once, breaking single-agent RL’s fixed-environment assumption.
• Non-stationarity is the core problem: from one agent’s view the environment changes as the others learn, so the best action is a moving target (the demo flips every episode) and naive single-agent RL fails to converge.
• Settings are cooperative (shared reward), competitive (zero-sum), or mixed; cooperative adds credit assignment — which agent earned the team reward?
• The dominant fix is Centralized Training, Decentralized Execution (CTDE): a centralized critic (sees all) stabilizes training and assigns credit; agents execute on local observations only — realized by MADDPG, QMIX, MAPPO.
• It powers AlphaStar/OpenAI Five and fleet/traffic coordination; LLM multi-agent systems that learn inherit the same non-stationarity and credit-assignment challenges.

Quick check

Next

MARL is what multi-agent systems face when agents learn rather than just coordinate; its agreement-theory cousin is consensus & Byzantine fault tolerance, and its reward machinery connects to RL alignment.