How does a robot learn to recognize an object it's never encountered? In this case, a demo can be worth a thousand words.
Short human demonstrations can be used to create fully automated training...
How does a robot learn to recognize an object it's never encountered? In this case, a demo can be worth a thousand words.
Short human demonstrations can be used to create fully automated training datasets, sidestepping the prompting limitations that hold back vision-language models. Rather than describing objects with language, the system tracks what a person touches and manipulates during a demo, follows those objects through time, and clusters detections to handle objects merging or splitting apart in the scene. This bypasses a core weakness of VLMs, which struggle to reliably detect unusual or novel objects even with repeated, carefully engineered prompts.
The result? Task-specific detectors that are trained in minutes with zero manual labeling and outperform state-of-the-art vision-language models on both detection accuracy and real-world task success. Down the line, this work can help folks who are domain experts in other areas - mechanics, assembly line workers, etc. - more easily program robots without needing to become robotics experts themselves.
Learn more about the “Show, Don’t Tell” approach here
Short human demonstrations can be used to create fully automated training datasets, sidestepping the prompting limitations that hold back vision-language models. Rather than describing objects with language, the system tracks what a person touches and manipulates during a demo, follows those objects through time, and clusters detections to handle objects merging or splitting apart in the scene. This bypasses a core weakness of VLMs, which struggle to reliably detect unusual or novel objects even with repeated, carefully engineered prompts.
The result? Task-specific detectors that are trained in minutes with zero manual labeling and outperform state-of-the-art vision-language models on both detection accuracy and real-world task success. Down the line, this work can help folks who are domain experts in other areas - mechanics, assembly line workers, etc. - more easily program robots without needing to become robotics experts themselves.
Learn more about the “Show, Don’t Tell” approach here
This website uses cookies
Comments 0
Be the first to comment!