Google DeepMind Simulates Robot Behavior Using Veo World Model

• DeepMind and Gemini Robotics used a video-based simulation system to test robot behavior across tasks without physical hardware.
• The system accurately predicted performance and flagged safety issues by simulating both standard and altered environments.

DeepMind and the Gemini Robotics team built a world model using Veo, Google's frontier video foundation model, to evaluate robot behavior without needing physical hardware. The model was fine-tuned to generate realistic, multi-camera predictions of future robot actions based on current scene images and planned motion sequences. It also includes generative tools for editing scenes across multiple axes, such as objects, backgrounds, and distractors, to support testing under varied conditions.

The system was used to test eight vision-language-action models on typical tasks as well as more difficult ones, where scenes were changed by adding new objects, visual clutter, or background shifts using Veo’s editing features. The evaluations were conducted on a bimanual robot and included instruction-driven tasks such as pick-and-place and object relocation, for example, placing red grapes into a compartment or putting a LEGO piece into a bag. Across more than 1,600 real-world evaluations, DeepMind reports the simulated results closely matched physical tests, showing the approach can rank and compare model performance with accuracy.

The team also used the system to red team safety by editing scenes with risky instructions or objects, revealing unsafe behaviors before using real hardware. This makes it possible to evaluate robot behavior more quickly, safely, and across a wider range of scenarios.

🌀 Tom’s Take:

DeepMind research shows that video simulations can accurately test how robots behave, fail, or act unsafely, making it easier to improve robotics without relying on costly or risky real-world trials.

Source: DeepMind