Google Genie Transforms Street View into Immersive Street Simulations

Most of us have utilized Google Maps Street View to revisit a childhood home or explore a potential hotel neighborhood in a distant city. Imagine extending this experience into a far more immersive and interactive simulation, allowing you to dynamically adjust environmental factors like weather or even visualize hypothetical scenarios, such as a "Day After Tomorrow"-esque landscape.

This advanced capability is a core objective of Google's latest integration. During the Google I/O developer conference, Google DeepMind unveiled a new feature connecting Street View to Project Genie, the company’s versatile world model designed to generate diverse, interactive environments.

“It’s really powerful for both the agent [and robotics] use case and for humans to play with, and that’s always been the thesis of Genie,” Jack Parker-Holder, a research scientist on DeepMind’s open-endedness team, shared with TechCrunch.

Parker-Holder provided an illustrative example: preparing a new robot for deployment in London, a city where sunshine is often scarce. Genie could simulate those infrequent moments when the sun illuminates Victorian architecture, ensuring the robot isn't startled by unexpected glare.

He further elaborated, “Simultaneously, you might say, ‘I’m going to New York City, but not this time of year. It’s going to be snowy. I want to see what that block looks like in the snow.’”

Google has been meticulously gathering Street View data for two decades, employing camera-equipped vehicles and individuals with "tracker backpacks." This extensive effort has amassed over 280 billion images, spanning 110 countries and all seven continents.

“With Street View, we have imagery from a large quantity of the world,” Jack remarked. “You can imagine how potentially powerful it is to combine this rich source of real-world information and data with an ability to simulate worlds.”

Google initially launched Genie 3, its most recent world model, for research preview last August. By January, Google AI Ultra subscribers in the U.S. gained access, enabling them to construct interactive game worlds from text prompts or images. The overarching aim for Genie includes applications in educational experiences, gaming, and robotics training.

Genie 3 is already instrumental in powering one of Waymo’s simulators, where it trains self-driving cars to navigate "exceedingly rare events" such as tornadoes or unexpected encounters with elephants. Integrating Street View data into this framework could significantly bolster Waymo's readiness for expansion into more global cities.

While Waymo operates its own simulator, crucial for its expansion into 11 U.S. cities and testing its AI driver in additional locations, Parker-Holder notes a key distinction. Waymo's existing simulations are exclusively from the car's perspective. Genie, leveraging Street View, not only allows for simulating environments anchored to real-world locations but also offers the flexibility to shift viewpoints to other agents, be it a human or a robot.

Starting today, Google is rolling out Street View integration within Genie to a select group of Ultra users in the United States, with broader access to follow. Global Ultra subscribers are expected to gain access over the coming weeks, according to the company.

Diego Rivas, a product manager at DeepMind, stated the researchers' ambition to make this new capability widely accessible. However, he cautioned that both Street View in particular and Genie in general remain experimental, indicating significant scope for accuracy improvements.

During a demonstration by the Google team, which included an impressive underwater simulation of a familiar neighborhood, the results were recognizable and engaging, yet presented with a visual quality akin to video games rather than photorealism. A current limitation is the models' lack of physics awareness, meaning they do not yet grasp cause and effect. For instance, in one simulation, a woman running through a snowy Joshua Tree landscape passed directly through cacti and bushes.

This contrasts with other advanced Google AI tools, such as the Nano Banana image generator, which can now produce flawless text within infographics, or the Veo video generator, which accurately understands physical phenomena like paper boats drifting on water currents, smoke dissipating, and fabric draping realistically over forms.

It's important to note that physics is not explicitly programmed into these models; rather, they acquire this understanding intuitively through passive observation over time, similar to how a living organism learns.

“I think for this kind of model, it’s maybe six to 12 months behind video in terms of the accuracy and quality, so I think it’s something we will solve,” Parker-Holder affirmed.

Jonathan Herbert, director of Google Maps, who began his career on the Street View team as an intern 12 years ago, acknowledged that Genie cannot yet achieve a perfectly faithful reconstruction of a street. He highlighted the AI’s spatial continuity as the genuine breakthrough: the ability to correctly remember and simulate the environment behind you when turning 360 degrees, and subsequently build new environments upon that foundation.

“We have long thought about how we can build out the best and richest model of the world on top of Street View data,” Herbert commented. “It’s definitely been an idea of ours to use Maps Data in new ways and for new kinds of AI research for a pretty long time.”