New Approaches to AI Training Using Video Game Data
With just a flick of a lever, a squeeze of a trigger, and the press of a few buttons, even novice players can navigate their way through a 3D video game with relative ease. A British startup is betting that such simple sequences hold a wealth of valuable information for training new artificial intelligence (AI) models.
Within certain sectors of the AI industry, there is a growing belief that large language models (LLMs) may ultimately be hindered by their inability to interact within the physical world. Since they are trained exclusively on text data, LLMs may not be equipped to drive autonomous vehicles, control robotic arms, or perform any action requiring finesse and precision. To address this gap, a group of renowned researchers, including Fei-Fei Li and Yann LeCun, is shifting their focus towards a different category of AI: world models.
Understanding Real-World Physics
To effectively grasp the physics of the real world, world models must be trained using a combination of visual data and actions. Before a model can skillfully manipulate a robotic arm, it may need to learn from video recordings taken within a factory setting, alongside information detailing how firmly it should grasp an object and the force required for manipulation, among other factors. Unlike LLMs, which are trained on vast corpora of text, labs working on world models lack a similarly extensive dataset.
“World models require an understanding of cause and effect,” explains Xiatian Zhu, an associate professor specialising in AI at the University of Surrey. “There is a significant shortage of such data available online.”
Utilising Video Game Data for AI Training
Worldmodeldata, a British startup advised by LeCun, aims to tackle this problem by collating control inputs and other data gathered from video game studies—an abundant by-product available in substantial quantities—into training datasets for world models. Some companies, such as General Intuition and Niantic, have already begun gathering data from their own gaming platforms to develop models. However, Worldmodeldata positions itself as an intermediary, streamlining the selection and organisation of data, thus sparing labs from negotiating individual agreements with countless gaming studios.
“There are millions of fantastic video games, and they increasingly resemble the real world,” states Rhea Loucas, CEO of Worldmodeldata. “Why not leverage the vast, abundant, and diverse experiences found in video games to teach AI?”
While this hypothesis has yet to be fully validated, researchers generally operate under the assumption that the performance of world models will improve in proportion to the size of their training datasets, akin to LLMs. Consequently, the scarcity of suitable training data poses one of the most significant challenges to progress.
Challenges of Manual Data Generation
Some labs have attempted to generate their own data manually by equipping individuals and robots with sensors in testing environments. However, this approach yields limited data and fails to account for the extreme scenarios that a model may encounter in the real world.
“You can pay people to demonstrate pick-and-place tasks. But repetition alone won’t capture the messiness of the world in which a machine is expected to operate,” notes Nicole Fraenkel, a partner at venture capital firm Khosla Ventures, which has invested in General Intuition.
The premise of Worldmodeldata is that the data collected from video game environments—3D visual representations combined with the actions performed by players—are both abundant and varied enough to encompass the critical extreme cases.
The Importance of Edge Cases in AI Training
“Exceptional cases are the ones that really need to be addressed correctly,” emphasises Fraenkel. “The cost of an error with a car, an aircraft, a drone, a factory forklift, or an autonomous quadruped robot is extraordinarily high.”
Worldmodeldata claims to have licensed nearly one million hours of data from developers of several popular video games, although Loucas has declined to disclose their names. Looking ahead, the startup aims to create pathways for individual players to also receive compensation for their contributions.
Loucas believes that video game data will ultimately constitute the majority of the training material for world models, which will then be optimised using specific data from real-world environments or tasks. “This could very well lead to the ‘GPT moment’ for world models, making them truly useful,” adds Loucas.
Concerns Over Video Game Data for Precision Tasks
Not everyone shares the same optimism regarding the potential of video game data. Nvidia, which releases a family of world models optimised for its chips, prefers to utilise a custom engine specifically designed to replicate real-world physics as the backbone of its AI.
Ming-Yu Liu, who oversees world model development at Nvidia, expresses doubts about whether models trained on video game data will excel in tasks requiring precise motor control, such as careful object manipulation. This scepticism stems from the fact that video game physics can often be eccentric; developers frequently take shortcuts to create the illusion of realism without programming intricate details, such as the pressure exerted by each finger to prevent an apple from slipping.
The Future of World Models and AI
“I would be more cautious about using video game data for manipulation tasks,” observes Liu. “The physics of manipulation is much more complex.” He believes that training with video game data is more suited for world models aimed at generating hyper-realistic videos or 3D environments.
Similarly, Zhu from the University of Surrey shares concerns about the limitations of video games. “Video games are, at their core, simulators. They have a degree of physical grounding,” he comments, “but they are very rudimentary and approximate.”
However, until world models reach their own breakthrough moment akin to that of ChatGPT, various ideas remain on the table for achieving this goal. “There are many paths to the promised land,” concludes Fraenkel. “The truth is, we still don’t know which will work best.”