The Waymo World Model (waymo.com)

Article summary

Waymo has introduced the Waymo World Model, a generative model for large-scale, hyper-realistic autonomous driving simulation. The model is built upon Google DeepMind's Genie 3 and can simulate rare events, generate high-fidelity multi-sensor outputs, and offer strong simulation controllability. This allows Waymo to safely scale its service across more places and new driving environments. The model can also convert videos into multimodal simulations, showing how the Waymo Driver would see a scene.

Main themes

• Autonomous driving simulation
• Sensor fusion
• Lidar technology
• Camera-only models
• Depth perception
• Human vision comparison

What commenters say

• Converting video to sensor data input can be used to train camera-only models, potentially allowing them to drive in conditions where lidar is not available.
• Human depth perception is not just based on stereoscopic vision, but also on focal distance, contextual clues, and other factors, making it challenging for camera-only models to replicate.
• Lidar is essential for providing accurate depth information, especially in conditions with low visibility, and its cost is decreasing, making it more feasible for widespread adoption.
• Some argue that operating in all conditions is not necessary, and that self-driving cars should focus on driving in all drivable conditions, while others emphasize the importance of being able to handle exceptional cases.
• The use of lidar can provide a significant advantage in certain situations, such as heavy rain or fog, where camera-only models may struggle to perceive the environment.
• The brain's ability to mash together multiple senses to make decisions is not yet fully replicable in autonomous vehicles, which may limit their ability to fully replace human drivers.
• The development of camera-only models is not mutually exclusive with the use of lidar, and some companies are using lidar to train and validate their camera-only models.
• The goal of autonomous driving is not to perfectly replicate human vision, but to create a system that can safely and effectively navigate the environment, which may require a combination of different sensors and technologies.