Robotics laboratories around the world are investing billions of dollars to generate the real-world data necessary to train their machines, addressing a critical shortage that is hindering the development of “physical AI” start-ups. Unlike traditional AI models, which rely heavily on extensive datasets scraped from the internet, robotics demands more complex and varied data, including video footage and sensory inputs that capture the physical nuances of tasks.

Companies such as Generalist, Physical Intelligence, Neura, and Skild are responding by partnering with external data providers and developing dedicated facilities known as “robot gyms.” These gyms are equipped with bimanual manipulators—robotic arms operated by humans—to perform physical tasks that robots must eventually emulate. Additionally, some start-ups are distributing recording equipment to thousands of homes, offices, factories, and holiday rentals globally. This equipment captures first-person, or “egocentric,” video footage of people conducting everyday activities, amassing millions of hours of training data.

“There is no internet for us to download from. There is no large corpus for model developers to work with,” said Joe Fox Jr, director of robotics operations at Scale AI, a key data supplier. “It is an arms race to get the best data.” According to PitchBook data, physical AI companies have raised nearly $48 billion in funding so far this year, much of which is being allocated to data generation infrastructure.

Ken Goldberg, a professor at UC Berkeley and co-founder of Ambi Robotics, explained that while large language models trained on web data can achieve impressive results, robotics presents a fundamentally different challenge. “It’s a completely new animal that needs to be trained from scratch with scarce data,” he said.

Several robotics firms are experimenting with innovative methods to obtain data. Physical Intelligence, valued at $5.6 billion after raising $600 million in funding last November, has tested its robots in various Airbnbs across San Francisco to explore model adaptability in different environments. Figure, a humanoid robotics company, has launched a program paying approximately 94 cents per task to around 44,000 users who record themselves performing household and workplace activities. The company plans to invest over $1 billion in the coming year to expand data collection and processing.

Aside from building in-house capabilities, many companies utilize third-party data providers. Firms like Encored, Scale AI, and others known for servicing large language model projects are now establishing “data factories” in locations such as the US, Mexico, China, and Indonesia. These facilities employ workers to annotate and label data, crucial for training robotic systems. Labor costs vary significantly; robotics operators earn roughly $25 per hour in the US, about twice the rate in Mexico, prompting some companies like Xdof, backed by Andreessen Horowitz and Thrive Capital, to shift operations to lower-cost hubs in Shenzhen and Jakarta. However, this outsourcing has raised ethical concerns, particularly in places like Gurugram, India, where factory workers are sometimes required to wear recording devices without additional compensation. An executive from a data-labeling company described conditions as “filming in literal sweatshops” and questioned the usefulness of such data.

Some robotics firms remain cautious about relying on external data, opting to handle most data generation internally to ensure quality. Nevertheless, the enormous volume of data required means that heavy investment in data collection is expected to continue. Goldberg noted that despite the risks, companies are wagering large sums on the potential payoff of these data-driven advances. “There’s no guarantee that if you generate a lot of data it will pay off,” he said. “Right now, it’s a hunch and many people are willing to pay for it.”