Omni & embodied models

The recipe behind every model in this chapter is the same: turn each modality into tokens, and predict the next token. Images joined text that way; the natural next question is — why stop there? Two frontiers extend the recipe further: omni models add audio, and embodied models add action, reaching into the physical world.

Omni: add audio, talk in real time

Omni models (GPT-4o, Qwen-Omni) bring audio into the same token stream — they understand speech and ambient sound, and crucially speak back, in real time. The common design is thinker-talker: a reasoning LLM “thinker” decides what to say, and a streaming speech “talker” turns that into audio as it’s produced, so the model can respond conversationally without waiting for a full text answer first. It’s the unified-token idea with audio added — one model that sees, hears, reasons, and voices a reply.

Embodied: action is just another token vocabulary

The bigger leap is embodied AI. A vision-language-action (VLA) model takes an image and an instruction and outputs robot actions — and the trick is that actions are just another token vocabulary the transformer predicts, exactly like words:

instruction = "pick up the red block"
action_tokens = ["move_x:+0.10", "move_y:-0.05", "gripper:close"]   # discretized motor commands

print(f"input:  image + instruction={instruction!r}")
print(f"output: {len(action_tokens)} action tokens ->")
for a in action_tokens:
    print(f"  {a}")

input:  image + instruction='pick up the red block'
output: 3 action tokens ->
  move_x:+0.10
  move_y:-0.05
  gripper:close

RT-2 and OpenVLA are built on a VLM, fine-tuned to emit discretized actions instead of text; π0 (pi-zero) uses a flow-matching action head for smooth continuous control. The profound consequence: because the action model is built on a web-pretrained VLM, the robot inherits internet-scale semantic knowledge — it can act on “pick up the extinct animal toy” because the underlying VLM knows which toy is a dinosaur, without ever being shown that mapping in robot data.

In one breath

• The token recipe (tokenize everything, predict the next token) extends past images to audio (omni) and action (embodied).
• Omni models (GPT-4o, Qwen-Omni) add audio in/out and respond in real time via a thinker-talker design (reasoning LLM + streaming speech).
• Vision-language-action (VLA) models output robot actions as tokens — RT-2/OpenVLA fine-tune a VLM to emit discretized actions; π0 uses a flow-matching action head for continuous control (the demo: image + instruction → action tokens).
• Because VLAs are built on web-pretrained VLMs, robots inherit internet-scale semantic knowledge (act on “the dinosaur toy” with no robot data teaching that).
• The grand unification: one architecture across text/image/audio/action — chatbot, voice assistant, and robot foundation model are becoming the same kind of model; embodied is the frontier because action data is scarce.

Quick check

Next

Omni and embodied models extend the unified-token recipe to audio and action, all built on VLM architectures. Embodied agents that act in software environments are covered in the Agentic AI track.