Beyond next-token: world models and the next paradigm

At 3:07 a.m., a warehouse robot reaches the end of an aisle. A pallet has been left half a metre too far out. The robot is moving, its battery is low, and a forklift is somewhere behind the rack.

Ask a language model what to do and it can give a sensible answer: slow down, check the left side, turn around, report the obstruction. It may even call the right tools. But a sentence is not a consequence. The robot needs to know what happens if it keeps moving for one second, brakes now, or turns with its current speed and wheel angle.

That is the gap world models are meant to close.

My view is that world models are not the replacement for language models that some headlines promise. They are a missing predictive component in the control stack: they enable a planner or controller to evaluate consequences before acting. An LLM is excellent at representing instructions, concepts, and human knowledge. A world model is useful when the important question is narrower and harder:

What will happen after this exact action, in this exact state, and how confident are we?

The distinction sounds simple. It is not. It changes what gets trained, what gets measured, and what counts as a convincing demo.

A world model predicts consequences

A language model learns a probability distribution over the next token, where a token is a small piece of text. Given “the cat sat on the”, it assigns high probability to “mat”. The training target is text continuation. That objective can produce impressive reasoning because text contains compressed descriptions of mathematics, physics, software, and human behaviour. The model is still predicting language, though.

A world model is a learned model of an environment’s changing state. A state is the information relevant to predicting what happens next. An action is something an agent can do to change that state. The central prediction looks like p(s_{t+1} | s_t, a_t): the likely next state given the current state and an action.

For the warehouse robot, the state might include:

• position and velocity;
• the map of nearby shelves and pallets;
• wheel angle and braking capability;
• battery level;
• the locations of moving obstacles.

A camera frame is an observation: sensor data generated by the underlying state. It may reveal only a partial, noisy projection of that state. Formally, one often writes o_t ~ p(o_t | s_t): the observation o_t is drawn from a distribution conditioned on the state s_t, rather than being the state itself.

The robot may see an empty patch of floor while a forklift is hidden behind a rack. A useful world model therefore needs memory or uncertainty. It should not quietly pretend that an incomplete image is the whole world.

Here is the difference with deliberately plain numbers. Suppose the robot is at position x = 2.0 metres along the aisle, moving at 0.5 metres per second toward a boundary at x = 2.4 metres, and the controller is considering the next one-second interval.

If it coasts, a simple motion model predicts x = 2.5 metres, which crosses the boundary. If it brakes with an acceleration of negative 0.4 metres per second squared, it predicts:

position = 2.0 + 0.5 × 1.0 + 0.5 × (-0.4) × 1.0² = 2.3 metres

Its predicted speed becomes 0.1 metres per second. In this toy example, braking keeps the robot short of an aisle boundary at 2.4 metres. The real controller would also model braking delay, body width, wheel slip, and uncertainty.

The calculation is not the point. The point is that the model is evaluating actions by their consequences.

An LLM can write this calculation. It can explain why braking is sensible. A world model is the component intended to estimate the outcome for this robot, on this floor, with this speed, after this action.

The boundary is not architectural. A world model can use a transformer, and an LLM can be connected to sensors and tools. “Next-token” describes a training objective, not a species of neural network. The important difference is whether the model is grounded in an environment and trained or adapted to predict its action-conditioned future.

That also means a world model does not need to reconstruct the entire universe. A model that predicts shelf geometry, collision risk, and wheel motion may be excellent for navigation while knowing nothing about the warehouse manager’s holiday schedule. “World model” is an ambitious name for a model that is usually useful only within a particular world, task, and sensor setup.

You can see why the transformer remains relevant. The same sequence machinery can process camera patches, actions, sensor readings, and latent state updates. The paradigm shift is from predicting what text comes next to predicting what the environment does next.

Prediction becomes useful when you plan inside it

A forecast alone does not make an agent intelligent. Planning does.

The robot can imagine three possible actions at each step: continue, brake, or turn. Over a five-step horizon, that produces 3^5 = 243 candidate action sequences. The world model rolls each sequence forward and predicts the resulting states.

The planner scores them using costs such as:

• collision risk;
• travel time;
• battery consumption;
• proximity to people.

The robot then chooses the best sequence, but executes only its first action. It observes the world again, updates its state, and plans again. This is receding-horizon control, often called model-predictive control: plan ahead, act briefly, check reality, and replan.

That last step is a safety feature, not an implementation detail. A learned model is never perfectly right. A five-step imagined route can be reasonable while the first turn is already wrong because the floor is wet or a wheel is stuck. Executing the entire imagined route open loop gives one early error time to become five later errors. Replanning keeps reality in the loop.

A world model also lets an agent ask counterfactual questions:

• What if I brake now rather than after the next sensor update?
• What if the pallet moves?
• What if the left passage is blocked?
• What happens if I spend battery on a longer but safer route?

Language models can discuss these questions in words. A grounded predictive model can attach them to measurable states and actions. That is the difference between a plausible explanation and a usable simulator.

There is another complication: the robot usually cannot observe the full state. If the camera sees the same empty corridor in two situations, but a forklift is hidden in one of them, the image alone does not determine the correct action. The model needs a belief state, which is a probability distribution over plausible hidden states.

It might represent “empty corridor” with 70 percent probability and “forklift behind rack” with 30 percent probability, then prefer an action that remains safe in both cases.

This is why a serious world model is more than a frame predictor. It needs several components:

• state;
• action;
• memory;
• uncertainty;
• often, a separate objective that says which futures are desirable.

The two camps: render the world or compress it

The current argument is not really about whether models should predict. Both camps agree on that. It is about what they should preserve.

The first camp says: render the world.

Video-generation systems such as OpenAI’s Sora are one way to model visual worlds as rich sequences. Sora’s documented interface conditions video generation on prompts and visual inputs; it is not, by itself, a documented robot-action-conditioned simulator. Google DeepMind’s Genie 2 is the interactive example: conditioned on a single image, it generates a playable 3D environment and predicts subsequent frames in response to keyboard-like actions such as moving or jumping.

The output may be:

• pixels;
• video frames;
• a manipulable visual environment.

This approach has an obvious strength. Images contain a great deal of information. A rendered future can show details such as:

• the pallet moving;
• a person stepping into the aisle;
• a cup falling from a table.

It does so without requiring engineers to define every object and rule by hand. It is also easy for humans to inspect. If the imagined scene turns into a nonsensical blur after two seconds, everyone can see the problem.

The weakness is equally important: visual plausibility is not causal correctness.

A generated video can show a robot turning smoothly while quietly changing the shelf layout. It can preserve the colour and texture of a car while losing the fact that the car occupies physical space. It can make a ball look as though it bounced correctly while failing when the agent tries the same action from a slightly different angle.

For control, the crucial question is not “does this frame look real?” It is “would the same action produce the same consequence in the actual environment?” A beautiful hallucination is still a hallucination.

The second camp says: compress the world.

Instead of predicting every pixel, an encoder maps observations into a latent representation, written as z_t. The model then predicts the next representation from the current one and the action. JEPA-style systems are an influential example of representation-space prediction: they learn useful embeddings without insisting on reconstructing every visible detail. Dreamer-style agents learn latent dynamics and use imagined trajectories to improve decisions.

Compression is not merely an efficiency trick. Most pixels are irrelevant to a particular decision. The warehouse robot does not need to predict the exact reflection of a fluorescent tube on the floor. It does need to preserve:

• shelf position;
• wheel slip;
• the possibility that a person is about to cross its path.

A good latent state removes nuisance detail while retaining variables that change the future. That is a hard requirement. If the encoder discards a wet patch because it looks visually unimportant, the latent model can be smooth, fast, and dangerously wrong.

The two philosophies are not mutually exclusive. A practical system may use a visual encoder, predict a compact latent state, decode selected futures for inspection, and render pixels only when visual detail matters. The useful distinction is not “pixels good, latents bad”. It is whether the representation preserves the information needed for the decisions the agent must make.

Recent systems have made world-model research more prominent, but deployment remains task-specific and reliability-limited. Interactive generation is being explored as infrastructure for agents, robotics, and simulated environments; research demonstrations are not the same as deployed robotics infrastructure. The serious test is closed-loop usefulness: can an agent act inside the model, encounter unfamiliar situations, recover from errors, and transfer what it learned to the real task?

The strongest objection is partly right

The strongest objection says that LLMs already contain a world model. They know that:

• dropped glasses tend to fall;
• doors have hinges;
• a robot should not drive through a wall.

Why add an expensive predictive system?

Because general knowledge and local dynamics are different kinds of knowledge.

A language model may know that braking reduces speed. It does not automatically know the current robot’s tyre friction, sensor delay, floor slope, or stopping distance. It may produce a reasonable plan from descriptions, but reasonable prose is not calibrated prediction. An agent operating in the physical world needs estimates tied to current observations and tested against consequences.

The objection is still correct in many software environments. If an agent is drafting an email, searching a stable database, or opening a reversible pull request, a language model with tools may be all you need. Building a learned simulator for those tasks adds:

• latency;
• training data;
• failure modes;
• a second system to operate.

There is no prize for replacing a reliable API with a speculative dream sequence.

Classical simulators also win when the rules are known. A physics engine or a hand-built warehouse simulator may be more accurate, interpretable, and cheaper than a learned model.

Learned world models earn their keep when the environment is:

• visually complex;
• partially observed;
• difficult to specify;
• changing faster than the rules can be maintained.

The practical rule is simple: use a world model when prediction of consequences is the bottleneck. Do not use one merely because the phrase sounds like the next stage of AI.

That boundary belongs in the agent architecture rather than in a marketing diagram. Let the language model interpret goals and communicate with people. Let a predictive model handle the local dynamics where its forecasts can be measured.

The failure you will see first

The first failure is usually not dramatic model collapse. It is a plan that looks right for three steps and then drifts into a wall.

Suppose the training logs contain many straight runs but very few sharp turns at speed. The world model learns an optimistic turning response. In simulation, the planner discovers a fast route around the pallet. On the real robot, the front wheel turns late and the body clips the shelf.

The first symptom is a gap between imagined and observed state: predicted position is still clear of the shelf, while the sensor log says the robot is already too close. Inspecting generated frames alone may miss the issue because they remain visually plausible.

Three mechanisms commonly cause this:

1. Compounding error. A consistent five-centimetre position bias per step can become a one-metre bias over twenty steps. Receding-horizon replanning limits the damage, but it does not remove it.
2. Action distribution shift. The model has seen normal commands, then the planner chooses an unusual sequence specifically because the model predicts it will work.
3. Model exploitation. The planner finds a loophole in the simulator. It chooses an action that scores well under the model’s mistaken rules and fails immediately in reality.

The remedy is not to demand prettier samples. Instead:

• compare predicted and observed states;
• track uncertainty;
• test unusual actions;
• put hard safety constraints outside the learned model.

A model should be allowed to suggest a route, not redefine what a collision means.

What I would do on Monday morning

Start with one decision, not a general-purpose “world simulator”. For the warehouse robot, choose “brake, turn, or continue when an aisle obstruction appears”.

Establish a baseline

Write down:

• the state variables;
• the available actions;
• the observation interval;
• the success criteria.

Include the things that make the decision difficult:

• speed;
• braking delay;
• hidden obstacles;
• battery.

Build a baseline before training anything. Under explicit assumptions, treat free space as independently certified to be unoccupied for the reaction window. Also assume obstacles there are static, pose and velocity estimates have known bounds, and braking and actuator limits are measured.

Add margins for:

• sensing and actuation latency;
• robot body clearance;
• uncertainty.

Brake when estimated stopping distance plus these margins exceeds that certified free space. Visible free space alone is not a safety guarantee: an independent collision monitor or safe-stop policy must handle occluded and dynamic regions, including the hidden forklift.

The rule may be dull, but it gives you a real comparison. Measure:

• collision rate;
• successful passage;
• route time;
• unnecessary stops.

Without a baseline, an impressive animation can hide a worse controller.

Collect the right data

Then collect trajectories as explicit transitions. Record each transition’s:

• observation;
• action;
• next observation;
• outcome.

Keep the action that was actually executed, not merely the command that software intended to send. Also record:

• actuator saturation;
• dropped sensor frames;
• interventions.

Those details are often where the model first becomes dishonest.

Train and evaluate

Train the smallest predictive model that can answer the chosen question. If the task is collision avoidance, predict:

• nearby geometry;
• velocity;
• collision probability.

Do this before attempting photorealistic video. If those variables cannot be labelled reliably, learn a latent representation, but test whether it preserves the information needed for braking and turning.

Evaluate the model and the policy separately. A model can have low average prediction error while missing rare obstacles. A policy can succeed on familiar layouts while exploiting a simulator on new ones.

Hold out entire:

• layouts;
• lighting conditions;
• obstacle configurations.

Do this rather than randomly shuffling nearly identical frames. The agent evaluation work matters here because success rate alone will not tell you whether the model is calibrated.

Deploy cautiously

Finally, deploy in shadow mode. Let the world model and planner make predictions while the existing controller remains in charge. Log where the imagined next state disagrees with the observed one.

Put these on dashboards:

• imagined and observed state disagreements;
• action distributions;
• uncertainty;
• intervention events.

This is the sort of detail covered by agent observability. Only then allow the new planner to control a narrow slice of traffic, with a simpler safety controller able to take over.

The next paradigm is not “next-token models are obsolete”. Language remains the best interface we have for:

• goals;
• explanations;
• accumulated human knowledge.

But an agent that must move through the world needs more than a description of consequences. It needs a model that can test them before acting.

The winning systems will probably combine both: language for what the agent is trying to accomplish, and a grounded world model for what its next action is likely to do. The important race is not toward prettier simulations. It is toward agents that can:

• imagine;
• act;
• notice when they were wrong;
• change course before the pallet wins.

Check your understanding