Omni & embodied models

The recipe behind every model in this chapter is the same: turn each modality (a kind of data, such as text, image or sound) into tokens, and predict the next token. Images joined text that way; the natural next question is — why stop there? Two frontiers extend the recipe further: omni models add audio, and embodied models add action, reaching into the physical world.

Omni: add audio, talk in real time

Omni models (GPT-4o, Qwen-Omni) bring audio into the same token stream. They understand speech and ambient sound, and crucially speak back in real time.

The common design is thinker-talker. A reasoning LLM, the “thinker”, decides what to say. A streaming speech module, the “talker”, turns that into audio as it’s produced. The model can then respond conversationally without waiting for a full text answer first. It’s the unified-token idea with audio added — one model that sees, hears, reasons, and voices a reply.

Embodied: action is just another token vocabulary

The bigger leap is embodied AI. A vision-language-action (VLA) model takes a camera image and a written instruction and outputs robot actions. The trick is that actions are just another token vocabulary the transformer predicts, exactly like words:

instruction = "pick up the red block"
action_tokens = ["move_x:+0.10", "move_y:-0.05", "gripper:close"]   # discretized motor commands

print(f"input:  image + instruction={instruction!r}")
print(f"output: {len(action_tokens)} action tokens ->")
for a in action_tokens:
    print(f"  {a}")

input:  image + instruction='pick up the red block'
output: 3 action tokens ->
  move_x:+0.10
  move_y:-0.05
  gripper:close

RT-2 and OpenVLA are built on a VLM (a vision-language model, one that reads images and writes text). They are fine-tuned (trained further on robot data) to emit discretized actions instead of text. “Discretized” means each motor command is rounded into one of a fixed set of values, like a word from a vocabulary. π0 (pi-zero) instead uses a flow-matching action head, a small module that generates smooth continuous motions rather than picking from a fixed set.

The profound consequence: because the action model is built on a web-pretrained VLM, the robot inherits internet-scale semantic knowledge — it can act on “pick up the extinct animal toy” because the underlying VLM knows which toy is a dinosaur, without ever being shown that mapping in robot data.

In one breath

• The token recipe (tokenize everything, predict the next token) extends past images to audio (omni) and action (embodied).
• Omni models (GPT-4o, Qwen-Omni) add audio in/out and respond in real time via a thinker-talker design (reasoning LLM + streaming speech).
• Vision-language-action (VLA) models output robot actions as tokens — RT-2/OpenVLA fine-tune a VLM to emit discretized actions; π0 uses a flow-matching action head for continuous control (the demo: image + instruction → action tokens).
• Because VLAs are built on web-pretrained VLMs, robots inherit internet-scale semantic knowledge (act on “the dinosaur toy” with no robot data teaching that).
• The grand unification: one architecture across text/image/audio/action — chatbot, voice assistant, and robot foundation model are becoming the same kind of model; embodied is the frontier because action data is scarce.

Quick check

Next

Omni and embodied models extend the unified-token recipe to audio and action, all built on VLM architectures. Embodied agents that act in software environments are covered in the Agentic AI track.