Could Brain Waves Unlock Physical AI's Future?

The vanguard of physical artificial intelligence finds its embodiment in a Jenga game played within a San Leandro, California warehouse.

This facility is home to Encord, a company specializing in data tooling for training AI models. Andrew Ceja, referred to as a "pilot"—the company's term for its robotic trainers—is meticulously extracting wooden blocks from a precarious tower. He wears a headset equipped with a camera to track his visual field, a common practice for collecting robot training data. Uniquely, this headset also integrates sensors that measure his brain waves as he carefully dismantles the structure.

Encord is among a burgeoning group of startups positing that the primary limitation for advanced humanoid and warehouse robotics will not be model architecture, but rather the acute scarcity of real-world physical training data. Instead of merely assisting robotics firms in managing their existing data, Encord is pioneering a business model centered on generating the data they currently lack.

The brain wave-monitoring headset worn by Ceja is a creation of Zander Labs, a German neuroscience startup. Zander Labs operates on the premise that measuring brain activity to infer mental states such as error, intent, and surprise can yield a more valuable dataset for model training. Encord's collaboration with Zander is presently a pilot program. Encord states its objective is to construct an initial brain wave-tagged dataset, process it through customer robotics models, and assess its impact on performance before determining whether to scale the initiative.

Lucas Gehrke, a Zander neuroscientist overseeing the project, suggests that the intensity of brain activity at various points during a task provides crucial indicators for model developers seeking to identify when their most sophisticated, high-effort models are required.

Vineeth Velmurugan, Encord’s head of robot learning, characterizes this endeavor as the "bleeding edge" in addressing the robotics data bottleneck. Velmurugan, an alumnus of OpenAI’s robot lab and Berkshire Grey, a prominent warehouse automation firm, joined Encord to establish the company's internal data-creation team.

Initially founded to assist companies in annotating data for machine-vision applications and evaluating models, Encord observed a shift. As their clientele—which Velmurugan notes includes numerous leading robotics firms he is not authorized to disclose—began implementing end-to-end learning for robotic manipulation tasks, executives recognized the necessity of producing training data themselves, rather than simply managing it. "The data simply does not exist," Velmurugan asserted.

The hypothesis that generative AI can replicate its success with chatbots for robots consistently encounters this fundamental obstacle. Large Language Models (LLMs) were trained on the vast corpus of the entire internet and beyond. Sourcing comparable raw material to educate neural networks about physical manipulation presents a significant challenge: self-driving car companies gather their own data, but this approach is difficult to scale. While training from video can be effective, it often lacks the fidelity inherent in real-world data. Velmurugan estimates that a dataset approximately five times the size of YouTube’s video corpus will be necessary to achieve a breakthrough—a scale that underscores why data generation itself has evolved into a business, transcending a mere research problem.

Companies developing robot "brains" are now leveraging two primary data sources: "Egocentric" video, captured by workers wearing cameras and frequently augmented with additional camera angles and metrics, and data gathered from remotely operated robots. Encord employs both methods, collecting egocentric data from multiple factories globally and utilizing its San Leandro facility to pioneer new modalities, such as brain waves, or to compile datasets for fine-tuning specific skills.

During a visit by TechCrunch, pilots were observed using leader-follower rigs—systems of paired robotic arms where one is directly controlled by a human operator and the other mimics its movements—to generate data for tasks like pouring coffee from a pot into mugs (a notably unstable process) and stacking poker chips. "Every humanoid company has asked us for these pieces," Velmurugan confirmed.

Storage racks within the facility held an array of items typically used for training manipulators for household tasks: cartons of artificial flowers in vases, books, plastic vegetables, kitty litter trays and scoops, and bundles of wires.

At another station, pilot Sofia Infante meticulously maneuvered robotic arms to plug and unplug Ethernet cables from the rear of a server—a task data center operators eagerly seek to automate, provided robots can achieve the requisite precision. Taking a turn at the controls, this reporter quickly understood the current limitations: robotic pincers are considerably less dexterous than human fingers and lack the degrees of freedom inherent in human arms.

Another novel data modality under development at Encord involves a set of sensors strapped to the forearm to detect electrical signals in muscles. Traditional video of human hands manipulating objects often fails to capture the entire hand. Velmurugan aims to construct a 3D representation of the hand's position at any given moment using these arm sensors, thereby fostering a more comprehensive understanding for AI models.

Encord's datasets are meticulously annotated with physical descriptions of the video content—for example, "right hand tightens bolt"—to assist LLM-based models in comprehending actions. Velmurugan estimates that this dense annotation style is approximately 100 times more valuable than "junky ego data" for training specific tasks, yet costs only 20 times more to produce, which represents a favorable return on investment, at least in theory.

However, "20 times more" still translates to substantial financial outlay, and therein lies the crucial distinction: scraping text from the internet, the method by which LLM developers built their models from sources like Stack Overflow and the broader web, incurred minimal cost for frontier labs. Generating physical training data, conversely, is not inexpensive, and this limitation defines the boundary of the "physical AI as LLM" comparison. This type of data must be actively manufactured, not merely collected, fundamentally altering the economics of developing these advanced models.

Velmurugan affirms that progress is being made. Encord's broad visibility across industry programs allows the company to observe both startups and frontier labs identifying effective and ineffective strategies for enhancing physical AI models. This advantageous position—interfacing with numerous robotics companies simultaneously—is also a core component of Encord's value proposition. It enables them to discern which data techniques are gaining industry-wide traction before any single customer can.

This ongoing demand ensures the continued activity of the dozen or so pilots at Encord's facility. Both Infante and Ceja are part of a burgeoning workforce dedicated to developing the foundational elements for neural networks; they previously worked at Scale, another AI data annotation firm, before joining Encord.

Ceja's prior experience at a waste management company led him to oversee the maintenance of a robotic trash sorter, where his interest in technology blossomed. Now, as the Jenga tower inevitably collapses, he expresses enjoyment in the challenge of devising training tasks for robots, remarking, "It’s something new every day!"