Exploring Robotic Innovation in Cambridge
Recently, I ventured a mere fifteen minutes from my home to witness robots performing astonishing tasks, leaving me utterly captivated. My destination was the offices of a startup named Generalist AI, situated in Cambridge, Massachusetts. There, I observed robotic arms engaged in simple tasks such as stacking cups and placing blocks into containers, demonstrating an impressive ability to learn quickly, reminiscent of human capabilities.
Impressive Adaptability of Robotic Arms
The robotic arms showcased their proficiency in various tasks after merely watching a brief instructional video—remarkably, without any specific training for each individual task. One particularly striking example involved a robot instructed to sweep a block into a container using a broom and dustpan. When the broom was removed, the robot ingeniously improvised, using the dustpan as a makeshift broom and tossing the block into the container.
In another instance, a two-armed robot observed a video of a person opening a wallet and retrieving banknotes. To my astonishment, the robot successfully opened a different type of wallet and delicately extracted the notes. When it struggled to grasp the money, it switched from its right to left gripper to achieve a better angle. “Wow,” exclaimed an engineer nearby, “I’ve never seen it do that before.”
Drawing Parallels with Human Learning
Co-founder and CEO Pete Florence remarked on the excitement surrounding these developments, likening them to the enthusiasm generated by OpenAI’s GPT-3, a groundbreaking large-scale language model released in 2020. “You could take that model and simply ask it to perform a new task, and it would have a strong chance of completing it,” he noted.
Generalist AI’s focus lies in teaching its robots the fundamental physics of the world, drawing inspiration from the intuitive grasp of physics that humans exhibit from a young age. This approach appears to enhance the model’s ability to transfer knowledge across different scenarios. Several demonstrations reminded me of how young children improvise and experiment when faced with a task. Researchers often find themselves surprised by the robots’ decisions, such as one robot opting to sweep objects with a banana when it was placed in front of it. While this may seem trivial, the physical intelligence exhibited by machines is still notably lacking, and understanding how infants learn efficiently could provide significant insights for AI researchers.
A Team with Remarkable Expertise
I had the pleasure of meeting Florence and Andrew Barry, the co-founder and Chief Technology Officer, in a meeting room overlooking teams training robots equipped with specialised grippers. The third co-founder and Chief Scientist, Andy Zeng, rounds out this impressive trio. They all boast an extensive background, having previously worked at Google DeepMind and Boston Dynamics on some of the most advanced hardware and robotics models available today.
A Shift Towards Generalist Robotics
Traditionally, training an AI-powered robot to perform various tasks has required thousands of examples to be fed into the model. However, this learning method is notoriously imperfect, often leading to difficulties with even minor changes in conditions, such as lighting. Generalist AI and other robotics startups are investing significantly in a generalist robotic model trained through human interaction. The company has developed special gloves resembling robotic grippers, equipped with cameras, which humans can use to perform different household tasks. I observed a box filled with hundreds of these grippers destined for workers in Mexico and beyond.
While Florence and his team remain somewhat secretive about the specific methods they employ to train the robots, they assert that the company has already amassed a substantial amount of high-quality training data. Unlike other companies pursuing smarter robots, Generalist has built its AI models entirely from scratch, rather than relying on an open-source language model.
Prospects for Real-World Applications
Danfei Xu, a robotics specialist from Georgia Tech familiar with Generalist’s work, notes that this startup stands out among those pursuing more general robotic models. “They’ve pushed this to the limit and executed it exceptionally well,” Xu remarks. In addition to gathering a vast amount of high-quality data, he adds, “they are excellent robotics specialists who have conducted truly impressive scientific work.”
Xu also highlights that Generalist’s progress thus far suggests they are aiming for deployment in real-world commercial environments. “They are the closest to something that can be implemented,” he asserts.
According to Karen Liu, a robotics expert from Stanford University who is also acquainted with the company, Generalist’s approach to data involves collecting large-scale physical interaction data without overly tying it to specific robots. “Their best results suggest that this strategy may be paying off,” she explains.
Challenges and Future Potential
Nevertheless, Generalist cautions that the learning capabilities of its models are still not entirely reliable. On average, a robot can complete a task it has been taught only about 59% of the time; ideally, this success rate should exceed 99%. It remains unclear how well these skills can be generalised across all conceivable tasks or environments.
Despite this, the potential for robots to rapidly acquire skills in manufacturing appears immense. One of Generalist’s engineers seemed to discover this during a recent afternoon. A video captured the moment he stacked small cups on a table in front of a two-armed robot, merely to observe its actions. Suddenly, the robot joined in, picking up and stacking additional cups with its grippers. As the robot completed the task, forming a neat stack, the engineer erupted in excitement.