RAI Institute RAI Institute
July 21, 2026

Show Don’t Tell: Teaching Robots to Detect Objects via Human Demonstration

How does a robot learn to recognize an object it's never encountered? In this case, a demo can be worth a thousand words.

Short human demonstrations can be used to create fully automated training...

How does a robot learn to recognize an object it's never encountered? In this case, a demo can be worth a thousand words.

Short human demonstrations can be used to create fully automated training datasets, sidestepping the prompting limitations that hold back vision-language models. Rather than describing objects with language, the system tracks what a person touches and manipulates during a demo, follows those objects through time, and clusters detections to handle objects merging or splitting apart in the scene. This bypasses a core weakness of VLMs, which struggle to reliably detect unusual or novel objects even with repeated, carefully engineered prompts.

The result? Task-specific detectors that are trained in minutes with zero manual labeling and outperform state-of-the-art vision-language models on both detection accuracy and real-world task success. Down the line, this work can help folks who are domain experts in other areas - mechanics, assembly line workers, etc. - more easily program robots without needing to become robotics experts themselves.

Learn more about the “Show, Don’t Tell” approach here
62

Comments 0

Replying to
5000 max

Be the first to comment!

This website uses cookies
This website uses cookies.
To maintain our services, we use cookies. They allow us to personalize ads, analyze our traffic, and provide social media features.

Customize your preferences

Select the cookies you accept: