Cornell Framework Teaches Robots from a Single Demonstration Video

• Robots using RHyME improved task success by over 50% with just 30 minutes of robot data.
• The framework draws on past video examples to bridge differences between human and robot actions.

A Cornell research team has developed a new training method for robots that relies on a single demonstration video. The approach, named RHyME, addresses a common challenge in robotic learning—how to manage discrepancies between human actions and robotic capabilities—by allowing robots to infer actions from related visual data they have previously encountered.

Instead of copying actions exactly, RHyME retrieves relevant snippets from past demonstrations and adapts them to fit the new task. When shown a person performing a sequence, like placing a mug in a sink, it draws from its video memory to find similar steps, such as grasping or lowering, and pieces them together to complete the task.

“This work is a departure from how robots are programmed today. The status quo of programming robots is thousands of hours of tele-operation to teach the robot how to do tasks. That’s just impossible,” said senior author Sanjiban Choudhury, assistant professor of computer science at the Cornell Ann S. Bowers College of Computing and Information Science, on the school's website. “With RHyME, we’re moving away from that and learning to train robots in a more scalable way.”

In lab experiments, RHyME-equipped robots performed complex sequences with over 50% higher task success than prior methods, using just 30 minutes of training data. The system's ability to handle multi-step tasks from a single demonstration highlights a shift in how robotic learning can operate under real-world conditions.

🌀 Tom's Take:

Teaching robots how to perform tasks that we take for granted as humans is a significant undertaking, and this research suggests that there are new, more scalable methods to explore, including many that we apply to our development.

Source: Cornell News