# What is a world model: AI that simulates what comes next

> World models predict how a scene changes when something acts in it, which lets robots, cars and game agents practice in simulation. Three papers from November 2026 show where the field stands.

URL: https://claudegpt.si/news/what-is-a-world-model  
Published: 2026-11-14T22:55:30.120Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: World models, Nvidia, Robotics, Video generation, Research papers  
Publisher: ClaudeGPT Si

## Key takeaways

- A world model predicts how an environment changes, given what it has seen and an action, so an agent can practice in a learned simulation.
- World models are used to train game agents, robots and self-driving systems, and to run games without a game engine, as in GameNGen's simulation of DOOM.
- NVIDIA's BirdDreams, inclusionAI's Duo and the WorldDraft benchmark, posted to arXiv November 10 to 12, 2026, push world models into reactive traffic, live conversation and rendering.

A world model is an AI system that predicts how an environment will change, usually in response to an action. Give it what it has seen so far and a control signal, such as "steer left" or "press fire", and it produces what comes next, often as video. That is the definition the [Duo paper](https://arxiv.org/abs/2611.15131) uses: given past observations and control signals, world models "predict how scenes evolve."

So what is a world model for? It lets an AI system try things out in a simulation it learned, instead of in the real world. A robot or a self-driving car can rehearse a move before making it, and a game can run without a game engine. Three papers on Hugging Face's daily papers list for November 13, 2026 show where the field is now.

## How a world model works

The idea is older than today's video models. In the 2018 paper ["World Models"](https://arxiv.org/abs/1803.10122), David Ha and Jürgen Schmidhuber trained a generative model of simple game environments. It learned a compressed picture of each environment and of how it changes over time. Then they trained a small agent, the part that picks actions, inside that learned simulation. They called it training "inside of its own hallucinated dream", and the agent's skills carried over to the real environment.

That loop is the core of the technique:

1. **Observe.** The model takes in frames, sensor readings or a compressed summary of them.
2. **Predict.** Given an action, it predicts the next state: the next frame, or a compact description of the scene.
3. **Imagine and plan.** An agent can run many predicted futures and pick the best action without touching the real world.

[DreamerV3](https://arxiv.org/abs/2301.04104), published by Danijar Hafner and colleagues in 2023, made this work across many tasks. It "learns a model of the environment and improves its behavior by imagining future scenarios", according to the paper. With one set of settings it beat specialized methods on more than 150 tasks, and it was the first algorithm to collect diamonds in Minecraft from scratch, without human data.

Newer world models predict pixels directly, using the same diffusion models that generate images and video. A diffusion model starts from noise and cleans it up step by step into a picture.

## What world models are used for

### Games

[GameNGen](https://arxiv.org/abs/2408.14837), a 2024 paper by Dani Valevski and colleagues, replaced the engine of the game DOOM with a diffusion model. The model draws each new frame from the earlier frames and the player's actions, and runs at 20 frames per second on one TPU. Human raters were only slightly better than chance at telling short clips of the real game from the simulation. Google DeepMind's [Genie 4](https://claudegpt.si/news/genie-4-world-model-preview), announced in October, extends this to interactive scenes with sound.

### Robots

NVIDIA's [Cosmos](https://arxiv.org/abs/2501.03575) paper from January 2025 makes the case for robots and other "physical AI". Such systems need "a digital twin of the world, the world model" to be trained digitally first, it says. Cosmos is a set of pretrained world models that developers fine-tune for their own setups, released as open-weight models with permissive licenses.

### Driving

A driving world model shows what a car's camera would see if the car took a different path. That lets engineers test rare, dangerous situations that are hard to stage on real roads, as the BirdDreams paper puts it.

## This week: driving, conversation and rendering

### BirdDreams teaches a driving simulator where cars are

[BirdDreams](https://arxiv.org/abs/2611.14815), from researchers at NVIDIA and KAIST, generates two videos at once from a single camera image: the driver's-eye view and a simple top-down map of roads and vehicles, called a bird's-eye view (BEV). The map shows positions at a fixed scale, so cars are less likely to hide behind each other. The two halves of the model share attention, so each video informs the other.

The point is traffic that reacts. Earlier models such as NVIDIA's OmniDreams took the other cars' paths as an input, replayed from driving logs, so those cars could not respond when the simulated car changed course. In BirdDreams, other vehicles respond: in one example, a following car slows down after the simulated car cuts in. The team trained it on about 670 hours of driving. Without being told where to drive, its paths stayed closer to the real recorded ones than every baseline tested. Its minADE, the average gap between a generated path and the recorded one (lower is better), was 6.15 on NVIDIA's PhysicalAI-AV clips against 7.76 for the strongest baseline, and 3.91 against 6.20 on nuScenes. In a human evaluation, participants preferred how its surrounding traffic behaved. The [project page](https://research.nvidia.com/labs/sil/projects/birddreams/) has video samples.

### Duo puts someone inside the world

Most world models simulate a place and leave the thinking to the user with a keyboard or a gamepad. inclusionAI's [Duo v1](https://arxiv.org/abs/2611.15131) adds a character that watches and listens to you in real time and answers with generated video and audio. One module decides, every 17 frames (about 0.71 seconds), whether to speak or wait. A second module generates the matching picture and sound. The authors call it a "Reciprocal World Model". They say it is preliminary and supports basic audiovisual conversation, and that response timing, long-session consistency and recovery from misunderstandings still need broader evaluation. The paper links its code on [GitHub](https://github.com/inclusionAI/Duo) and a model collection on Hugging Face.

### WorldDraft tests video models as renderers

[WorldDraft](https://arxiv.org/abs/2611.16217) asks a different question. If a coding agent builds a scene in a simulator such as Blender or MuJoCo, can a video model just make it look real without changing what happens? The benchmark has 432 paired test cases across 48 scene families and tested 12 models. Keeping the motion intact and getting the requested look turned out to pull against each other. More detailed drafts held the structure but hurt the appearance, and when a draft's colors conflicted with the prompt, source leakage, where the draft's original look shows through, reached 64%. The authors propose a prompting fix, WorldDraft-Primer, that improved both scores on the models they tried.

## What world models still get wrong

All three papers measure their models against human judges or narrow benchmarks, not real-world deployment. WorldDraft found that a video can look real and still break the scene underneath it. A [survey we covered in October](https://claudegpt.si/news/video-world-model-survey) makes the same point at length: smooth video is not proof that the model keeps a consistent world. Until world models hold up over long sessions, treat them as useful simulators for testing and training, not as faithful copies of reality.

## Frequently asked questions

### What is the difference between a world model and an LLM?

A world model is built to predict how an environment changes when an action is taken, often as video frames or a compact scene description. A large language model predicts text. Some systems combine the two: Duo v1 uses a multimodal language model module to decide when to speak and a diffusion module to generate the video and audio.

### What is a world model for autonomous driving?

It is a model that learns from real driving video to show what a car's camera would see if the car took a different path. NVIDIA's BirdDreams generates both the camera view and a top-down map from a single image, so surrounding traffic can react to the simulated car.

### How are world models used in robotics?

They let a robot's control policy be trained and tested in a learned simulation before it runs on hardware. NVIDIA's Cosmos paper describes world models as a digital twin of the world for this kind of physical AI.

### Are there open-source world models?

Yes. NVIDIA released Cosmos as open-weight models with permissive licenses, and the Duo v1 paper links its code on GitHub and a model collection on Hugging Face.

## Sources

- [Duo: A Reciprocal World Model for Full-Duplex Interaction](https://arxiv.org/abs/2611.15131), arXiv (inclusionAI)
- [BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model](https://arxiv.org/abs/2611.14815), arXiv (NVIDIA, KAIST)
- [BirdDreams project page](https://research.nvidia.com/labs/sil/projects/birddreams/), NVIDIA
- [WorldDraft: If Code Builds the World, Can Video Models Just Render It?](https://arxiv.org/abs/2611.16217), arXiv
- [World Models](https://arxiv.org/abs/1803.10122), arXiv
- [Mastering Diverse Domains through World Models](https://arxiv.org/abs/2301.04104), arXiv
- [Diffusion Models Are Real-Time Game Engines](https://arxiv.org/abs/2408.14837), arXiv
- [Cosmos World Foundation Model Platform for Physical AI](https://arxiv.org/abs/2501.03575), arXiv (NVIDIA)
- [Daily papers, November 13, 2026](https://huggingface.co/papers/date/2026-11-13), Hugging Face
