Aditya Ramabadran, Simon Mahns, and Tobias Gessler, three AI engineers from a startup named Axiom, recently found themselves craving a meal from In-N-Out Burger. However, they had no intention of driving there themselves.
Sitting in a 2024 Toyota Corolla near the drive-thru of the Bay Area restaurant, they opened a laptop and requested GPT-6 Astra, an AI model from OpenAI, to take the wheel. They connected a chat interface to a server, which was in turn linked to several cameras mounted on the windshield and the car’s assisted steering system. A safety driver kept a foot on the brake, just in case.
The AI model, typically responsible for generating text, code, and occasional images, navigated the vehicle slowly but safely to the window, allowing them to collect their food. “Perhaps general artificial intelligence (AGI) is already here after all,” commented one of the engineers, alluding to the popular concept of machines capable of mimicking human intelligence.
Exploring New Frontiers in AI
While autonomous vehicles are not a novel concept, they are generally controlled by algorithms specifically trained for that task. In this instance, however, the operation was executed by a model designed for text generation, and it was conducted spontaneously, without any prior training from the trio. This achievement at the fast-food restaurant, along with other experiments, suggests that language-based AI models are beginning to develop a rudimentary but useful understanding of the physical world.
Despite the uneventful nature of their outing, the engineers acknowledged that placing a general-purpose model in charge of a two-ton steel block moving at high speeds is inherently risky. As our grasp of physics improves, we may witness how AI models could integrate into the real world in new, astonishing, and potentially perilous ways.
The Challenges of Physical Reasoning
Today’s most advanced models excel at answering complex questions and even performing certain virtual tasks autonomously, yet their capabilities tend to be confined to the realm of internet-connected computers. When we attempt to apply these models to the real world, they often become bewildered. Despite claims that AGI has arrived, AI companies clearly view physical reasoning as an uncharted frontier.
Some researchers have left large corporations to establish startups focused on addressing the challenges of physical reasoning. Andrew Dai, the CEO of one such company, Elorian AI, previously worked as a researcher at Google DeepMind. He asserts that improved visual reasoning will unlock a host of new applications for AI, including systems that can determine whether diners are enjoying their meals in a restaurant or robots capable of functioning in a home environment. Dai emphasises that robotics serves as a crucial testing ground for physical reasoning abilities: “It’s essential; one cannot envision domestic robotics without it.”
Pioneering New Standards for AI Models
Elorian and Scale AI, a company that provides training data to major AI laboratories, have created a new benchmark, “Humanity’s Sixth Sense,” to measure models’ abilities to comprehend physical scenes. “Most research in visual AI focuses on perception. We wanted to investigate what it would actually take for a model to intuitively understand a scene, just as a person does without overthinking it,” explains Xingang Guo, a research scientist at Scale AI involved in developing the benchmark.
For Ramabadran, Mahns, and Gessler, the aim of their project, conducted alongside their work at Axiom, was to assess how effectively models could navigate the real world. The idea to test the driving capabilities of these models emerged during a weekend spent together. They realised that models like Astra were capable of creating complex 3D simulations and wondered if that could translate into real-world navigation.
Innovative Experiments with Driving Models
<p“All of us live in the Bay Area and are accustomed to seeing Teslas and Waymos, so perhaps that influenced us,” Ramabadran shares. The engineers initially tested the Grok model from SpaceXAI before experimenting with OpenAI's and Anthropic's latest models. At first, the models were reluctant to take control of the vehicles, responding with comments such as, “I can help interpret road images, but I cannot command a physical car to move.” However, with careful prompting, they were persuaded to go further. The trio was astonished when the models began to drive.
The engineers believe that major AI firms are currently focused on enhancing the spatial reasoning capabilities of their latest models, yet they question whether this truly equates to training them for driving. They suspect the models’ driving abilities may have emerged as a byproduct of training geared towards 3D reasoning. “This could be an emergent capability simply resulting from expanding the model’s multimodality,” Mahns explains, referring to the input data such as images, videos, and 3D models that are now being used to train the models.
Assessing the Road Ahead for AI Models
Nevertheless, a new driving benchmark developed by the trio, dubbed DrivingBench, indicates that these models still have a considerable way to go before they can pass a driving test. The assessment measures a model’s ability to navigate a simple circuit simulated in a parking lot. Only Astra was able to complete the course, and that was at a very slow pace. Claude Fable 5.1 managed to cover 45% of the route, while Grok only reached 11%.
Ramabadran notes that the latest models appear capable of adapting to mistakes; they seem to adjust to the vehicle’s controls and improve their driving in real-time: “The models genuinely seemed to be aligning or learning contextually from their errors and mastering the controls.”