Conference:  Transform X 2022

Authors: Mostafa Rohaninejad, Ariana Eisenstein, Louis Tremblay, Jack Guo, Russell Kaplan

2022-10-19

Machine learning leaders from robotics (Covariant), home automation (Resideo), autonomous delivery (Nuro), and warehouse automation (Pickle Robot) sit down with Russell Kaplan, Scale’s Director of Engineering, to share their approaches to dataset management. Pickle Robot CTO Ariana Eisenstein will share how she thinks about modulating quantities from different data sources like synthetic and public open datasets with real-world data for training datasets. Mostafa Rohaninejad, Founding Research Scientist at Covariant, will describe how the object “picking” problem requires synthetic data for unsafe scenarios and how he also incorporates structured and time-series data—supervised and unsupervised learning should go hand-in-hand. Jack Guo, Head of Perception at Nuro, will explain how it’s essential to have tools and mechanisms to automatically highlight recorded data that deviates from the norm, especially if it was captured in a new location. Like Rohaninejad, he will stress the importance of simulation as a component of successful reinforcement learning. Louis Tremblay, AI/ML Engineering Leader at Resideo, will explain how security cameras in the home represent an even more unbounded environment than do warehouses. The group will also discuss why maintaining separate datasets and training pipelines for different customers is both costly and incurs additional technical debt over time. Testing on fault-tolerant customers first before deploying to the wider fleet is also important. Scale’s Kaplan will share how, in his experience, when metrics and anecdotes seem at odds, it makes sense to re-think the metrics and establish new ones.

Conference:  Transform X 2021

Authors: Marc Segura

2021-10-07

The speaker discusses the importance of breaking down robotic skills into smaller skills and using AI to develop and improve them over time. The bottleneck for scaling robotics is the matching of hardware with software.

• Robot skills should be broken down into smaller skills for better learning and improvement over time
• AI can be used to develop and improve robot skills
• Hardware and software matching is the bottleneck for scaling robotics

Conference:  Transform X 2021

Authors: Fei-Fei Li

2021-10-07

The presentation discusses the creation of a large-scale and diverse robotic learning simulation environment and benchmark called Behavior Benchmark for everyday household activities in virtual interactive ecological environments.

• The goal is to create a robotic learning simulation environment and benchmark that mimics the real world as much as possible
• The environment is large-scale and diverse, with a thousand activities or tasks, 50 large-scale real-world things, 8 scene types, more than 2000 object categories, 3000 object models, and a large number of objects per activity
• The tasks are complex, like long horizon and multi-steps, and have standardized and flexible evaluation metrics
• The approach is human-centered, taking all the potential tasks that people do through surveys like American Bureau of Labor Statistics and Eurostats
• The simulation environment has photorealistic rendering, kinematic and dynamic extended states, flexible materials, deformable bodies, realistic fluids, thermal effects, realistic action executions, and object distributions