Reinforcement learning foundations

At 3 a.m., a warehouse robot has to choose between two corridors. One is short but often blocked. The other is longer but usually clear. Nobody has labelled the correct answer for every possible battery level, package, blockage, and traffic pattern. The robot must try something, observe what happened, and gradually learn which choices lead to deliveries.

That is the problem reinforcement learning solves.

The robot is not shown an answer. It receives a number after acting. A successful delivery might earn +10; wasting a minute might cost -1. The useful consequence may arrive much later, after several decisions. Worse, the robot’s choices determine which situations it encounters next. It can learn to avoid a corridor so thoroughly that it never gathers the evidence needed to judge it.

Reinforcement learning, usually shortened to RL, is learning a decision-making strategy from interaction and rewards rather than from labelled examples.

The loop: observe, act, receive

An agent is the learner that chooses actions. An environment is everything it interacts with: the warehouse, game, simulator, or language-model feedback process. At each time step, the loop is:

1. The agent sees a state.
2. It chooses an action.
3. The environment transitions to a new state.
4. The environment returns a reward.

The agent repeats this until an episode ends. An episode is one complete attempt, such as one delivery run or one game.

This differs from supervised learning in three important ways:

There are no action labels. A supervised example pairs an input with a known answer. In RL, “turn left” is not labelled as correct before the robot tries it.

Feedback is evaluative and often delayed. A reward says how desirable an outcome was, not which preceding action caused it. If the robot receives +10 only at delivery, earlier turns must share credit.

The agent controls its data distribution. An action changes the next state and therefore the next observation. A timid policy may collect only easy experiences; an adventurous one may collect useful evidence and also crash.

The agent’s decision rule is a policy, written π(a | s): a rule or probability distribution for choosing action a after seeing state s. A stochastic policy might assign π(left | s) = 0.3.

The Markov decision process

The standard mathematical model is a Markov decision process, or MDP. “Markov” means that the current state contains enough information to predict what matters next; the whole history is not required.

An MDP specifies:

• States, the situations the agent can be in.
• Actions, the choices available to it.
• Transitions, the probabilities of reaching each next state after an action.
• Rewards, the numerical feedback produced by a transition.
• Discount factor γ, pronounced gamma, which controls how much future rewards count.

Our warehouse robot can use a tiny grid:

• S is the starting square.
• A is one square from the goal.
• G is the delivery square.
• Moving from S to A costs -1.
• Moving from A to G earns +10.
• The episode ends at G.

The state must include anything that changes the future, such as whether a door is open or how much battery remains.

The return from time t is the total future reward, discounted by distance:

G_t = r_{t+1} + γr_{t+2} + γ²r_{t+3} + ...

Suppose the robot takes three transitions with rewards -1, -1, and +10, and γ = 0.9:

G_0 = -1 + 0.9(-1) + 0.9²(10) = -1 - 0.9 + 8.1 = 6.2

Gamma does not alter the reward emitted by the environment. It alters the agent’s preference for waiting. With γ = 0, the robot cares only about the next reward. With γ = 0.9, a reward ten steps away is weighted by 0.9¹⁰, about 0.35; with γ = 0.99, it keeps about 0.90 of its weight.

Values and the Bellman idea

A value function predicts future return. Under policy π:

V^π(s) = E_π[G_t | s_t = s]

An action-value function, or Q-function, is more specific:

Q^π(s, a) = E_π[G_t | s_t = s, a_t = a]

V asks, “How good is being here?” Q asks, “How good is taking this action here?”

The Bellman equation is a consistency condition: a value equals the immediate reward plus the discounted value of what comes next.

V^π(s) = E_π[r_{t+1} + γV^π(s_{t+1}) | s_t = s]

For the best possible behaviour, choose the best continuation:

Q*(s, a) = E[r + γ max_a' Q*(s', a') | s, a]

This Bellman optimality equation turns a delayed objective into a sequence of local predictions. An action can be judged partly by the estimated value of the next state.

Tabular Q-learning, by hand

Tabular Q-learning stores one number for every state-action pair. It is practical when there are only a few states, such as our three-square corridor.

The update is:

Q(s, a) ← Q(s, a) + α [r + γ max_a' Q(s', a') - Q(s, a)]

Here α is the learning rate. The bracketed term is the temporal-difference error, or TD error: the difference between a new one-step target and the old estimate.

Start with every Q-value at zero. Let α = 0.5 and γ = 0.9.

First, the robot is at A and moves right into the terminal goal:

• Old Q(A, right) = 0
• Reward r = +10
• There is no future value after a terminal state
• Target = 10
• New value = 0 + 0.5 × (10 - 0) = 5

Now the robot is at S and moves right into A:

• Old Q(S, right) = 0
• Reward r = -1
• Best known value at A is 5
• Target = -1 + 0.9 × 5 = 3.5
• New value = 0 + 0.5 × (3.5 - 0) = 1.75

The goal’s value has travelled one square backward. Repeated experience propagates it through the corridor.

q = {
    "S": {"right": 0.0},
    "A": {"right": 0.0},
}

alpha = 0.5
gamma = 0.9

def update(state, action, reward, next_state, terminal):
    old_value = q[state][action]
    if terminal:
        target = reward
    else:
        target = reward + gamma * max(q[next_state].values())
    q[state][action] = old_value + alpha * (target - old_value)

update("A", "right", 10.0, None, True)
update("S", "right", -1.0, "A", False)

print(f"Q(A, right) = {q['A']['right']:.2f}")
print(f"Q(S, right) = {q['S']['right']:.2f}")

It prints:

Q(A, right) = 5.00
Q(S, right) = 1.75

Q-learning is off-policy: the action used to collect data need not be the action assumed by its target. The robot can explore randomly while the update still asks what the best next action would be.

This creates the central RL tension: exploration versus exploitation. Exploitation chooses the action with the highest current Q-value; exploration gathers information about uncertain actions.

The simplest rule is epsilon-greedy. With probability ε, choose a random action; otherwise choose the action with the largest Q-value. Training often starts with a higher epsilon and lowers it. During evaluation, exploration is usually removed or made very small.

When a table becomes a neural network

A table cannot hold a useful entry for every camera image or board position. Function approximation replaces it with a model that generalizes across states.

A Deep Q-Network, or DQN, uses Q_θ(s, a) to estimate action values. Given a state, the network usually outputs one Q-value per discrete action, and the agent selects the largest output unless epsilon-greedy exploration intervenes.

The network is trained toward this TD target:

y = r + γ(1 - done) max_a' Q_θ'(s', a')

For a terminal transition, there is no future value, so y = r. The prime denotes a separate target network. The loss compares Q_θ(s, a) with y, often using squared or Huber-style error.

Using one changing network for both sides is unstable. The target moves whenever the same parameters change, so the model is chasing its own shifting answer. Also, consecutive transitions are correlated: a robot’s experiences at times 100, 101, and 102 are usually near-duplicates.

DQN uses two fixes:

• A replay buffer stores transitions (state, action, reward, next state, done). Random mini-batches reduce sequential correlation and let one experience be reused.
• A target network is a delayed copy of the online network. Its parameters θ' remain fixed for a while and are periodically copied from θ, making the target less volatile.

Replay changes the data mixture; the target network slows the target’s movement.

DQN still has limits. Replay can preserve obsolete experiences after the environment changes, while an infrequently updated target network can slow learning. Neural networks may also generalise confidently to barely explored states. A smooth loss curve does not prove that the policy is safe.

Learning the policy directly

Q-learning learns values and derives actions from them. Policy-gradient methods parameterize the policy itself, often as a neural network producing action probabilities. They adjust parameters θ to increase expected return.

The basic REINFORCE estimator is:

∇θ J(θ) ≈ Σ_t ∇θ log πθ(a_t | s_t) G_t

If an action appears in a high-return episode, increase its probability; if it appears in a poor-return episode, decrease it. ∇ log π gives the direction, while G_t determines the credit or blame. REINFORCE is simple but noisy, because a lucky outcome can make every earlier action look good. A baseline reduces this variance. An advantage A_t says how much better the action was than the policy’s usual expectation in that state.

PPO, or Proximal Policy Optimization, makes policy updates more conservative. It compares the new policy with the old one using:

r_t(θ) = πθ(a_t | s_t) / πold(a_t | s_t)

Its clipped objective is commonly written:

E[min(r_t(θ)A_t, clip(r_t(θ), 1 - ε, 1 + ε)A_t)]

For a positive advantage, increasing an action’s probability helps only up to the clip range. For a negative advantage, clipping limits further decrease. This reduces destructive updates caused by an unusually good or bad batch.

PPO clipping is not a safety guarantee or proof that two policies are close in every meaningful sense. Rollout length, advantage estimates, reward scale, entropy, and the environment still matter.

The first failures you will see

Sparse reward

A sparse reward arrives rarely, perhaps only when the robot reaches G. Before then, every transition may produce zero, leaving little information about useful actions. With random exploration, reaching a goal ten steps away may be so unlikely that learning never sees a successful trajectory.

A return plot pinned at zero and nearly identical Q-values across actions are common symptoms. Possible remedies include carefully designed reward shaping, curriculum tasks that begin near the goal, demonstrations, or exploration methods for long-horizon problems. Shaping must preserve the real objective; otherwise the proxy becomes the objective.

Reward hacking

Reward hacking occurs when the agent maximizes measured reward while violating its intent. Suppose the warehouse reward grants +1 whenever the robot passes a charging square but gives no penalty for repeating the circuit. The robot may loop forever instead of delivering the package. Its reward rises while the business result worsens.

A widening gap between reward and an independent outcome metric is the warning sign: reward improves while delivery rate, constraint violations, or human ratings deteriorate. Fixes include better reward design, terminal conditions, explicit constraints, adversarial tests, and monitoring the actions that earn reward. More reward terms can create a more elaborate loophole.

These failures follow from the setup. RL optimizes the reward it receives, not the engineer’s intent, and cannot learn from a consequence it almost never experiences.

What to remember

• RL learns a policy through an agent-environment loop: act, observe, receive reward, repeat.
• An MDP specifies states, actions, transition dynamics, rewards, and discount factor gamma.
• The Bellman equation turns delayed return into immediate reward plus discounted continuation value. Tabular Q-learning updates toward that target.
• DQN replaces tables with a neural network and uses replay plus a delayed target network to reduce correlation and moving-target instability.
• Policy gradients adjust action probabilities directly. REINFORCE is noisy; PPO limits individual policy updates.
• Sparse rewards starve learning. Reward hacking optimizes the score while missing the intent, so measure real outcomes separately.

This machinery is the foundation beneath later topics. RLHF uses reward models and policy optimization; DPO takes a different route to preference alignment; verifiable-reward training supplies checkable signals; and multi-agent RL extends the loop to several interacting decision-makers. On datarekha, those paths are /gen-ai/alignment-rlhf/, /gen-ai/dpo/, /gen-ai/reward-hacking/, and /agentic-ai/marl/.

Quick check

0/3

Q1

It guarantees that every sampled action is optimal.It randomizes stored transitions, reducing sequential correlation and reusing varied experience.It increases every reward before the neural network sees it.It removes the need for a target network.

Check answer

Q2

0.501.753.505.00

Check answer

Q3

The agent will always exploit the exit; lower gamma immediately.The agent may loop over the coin; inspect the reward and add suitable termination or task-completion signals.The agent cannot explore; set epsilon to zero.The agent has a moving-target problem; replace the reward with a larger neural network.

Check answer