🤖 iTeach

In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception

Jishnu Jaykumar P  ·  Cole Salvato  ·  Vinaya Bomnale  ·  Jikai Wang  ·  Ayush Bhardwaj  ·  Jin-Ryong Kim  ·  Yu Xiang

_{Intelligent Robotics and Vision Lab · The University of Texas at Dallas}

_{A pretrained perception model fails in the wild. A co-located human performs a short HumanPlay, annotates a single frame with eye-gaze + voice, and the label is propagated across the RGB-D clip. Failure-driven samples feed an iterative fine-tuning loop, and the best checkpoint is redeployed.}

iTeach adapts robot perception while the robot is deployed, triggered by its failures, instead of through offline retraining.

When the perception model fails, a human spends a few seconds rearranging the objects (HumanPlay, 5–10 s) while the robot records RGB-D. The human then labels the final frame hands-free with eye-gaze and voice on a HoloLens 2. SAM2 turns those point prompts into masks and propagates them backwards through the clip, producing dense supervision. That data is used to iteratively fine-tune an MSMFormer unseen-object instance segmentation (UOIS) model, which in turn improves grasping and pick-and-place.

🎯 UOIS combined score	🦾 Grasping success	📦 Pick & place success	👥 User study
26.1 → 80.7 +54.6 · 3.1× lift	71 → 74 / 100 +3 on SceneReplica	65 → 72 / 100 +7 on SceneReplica	94.9% mean box IoU NASA-TLX 21/100 · 12 participants

📑 Contents

🗂️ Full index _{(every section and subsection as text links)}

1. How It Works · The overview video and the FS3 labelling steps
2. Results · Perception, manipulation and the user study
   • Perception
   • Manipulation
   • User study
3. Code · Which repository does what, and in what order
   • Data & checkpoints
   • Hardware
4. Legacy · iTeach v1 · Archived door and handle detection (DH-YOLO)
   • Getting started
   • Directory structure
5. License · Citation · Contact · Thanks

---

🎬 How It Works

_{▶ Watch the overview video}

Labelling uses FS3 (Few-Shot Semi-Supervised): one human-labelled frame becomes dense supervision for the whole clip.

① HumanPlay: clean up the scene

_{The human rearranges objects to reduce occlusion and produce a clean final frame while a short 5–10 s RGB-D sequence is recorded.}

② Annotate hands-free: eye-gaze + voice

_{Eye-gaze places point prompts on the final frame; a voice command triggers SAM2 to convert them into bounding-box object labels.}

③ Propagate: SAM2 video mode

_{SAM2 propagates masks backwards from the final annotated frame to all earlier frames, producing dense per-frame supervision.}

_{⬆ back to contents}

---

📈 Results

Perception improves · Manipulation follows · Anyone can teach

🎯 Perception adaptation

UOIS combined score	Before	After	Gain
MSMFormer → iTeach fine-tuned	26.1	80.7	+54.6 · 3.1×

_{Qualitative UOIS. Left → right: ground truth, pretrained MSMFormer, and iTeach fine-tuning rounds FT1, FT3, FT5. iTeach recovers missed instances and cleans up over-segmentation across tabletops, shelves, sofas, stairs and floor-level scenes.}

🦾 Manipulation follows

Success / 100 (SceneReplica)	Prior best (MSMFormer)	iTeach-UOIS	Gain
🦾 Grasping	71	74	+3
📦 Pick & place	65	72	+7

_{Only the segmentation stage changes; everything else is held fixed. Swap in iTeach-UOIS and the same real-robot pipeline (SceneReplica: UOIS → Contact-GraspNet grasps → GTO motion planning) starts handling clutter and unseen objects the pretrained baseline fails on. How to run →}

👥 Who can teach? · 12-participant user study

👥 Participants	✍️ Annotations	🎯 Mean box IoU	🧠 NASA-TLX
12 6 expert · 6 non-expert	240 12 people × 20 objects	94.9% SD 0.49 · gaze + voice prompts	21 / 100 low task load

_{Each participant ran one full teaching interaction unassisted: spot the failure in the headset, perform HumanPlay, then label the final frame with gaze and voice. Expertise made no difference on any measure.}

_{⬆ back to contents}

---

🧩 Code

The current iTeach system is split into three repositories, one per module. Follow them in this order:

Step	Repository	What it does
1	🥽 iTeachSkillsApp	HoloLens 2 app + ROS bridge. Watch live predictions, record a HumanPlay clip, place point prompts with gaze + voice, get SAM2 boxes.
2	🧠 iTeach-UOIS	Propagate SAM2 masks backwards through the clip, fine-tune MSMFormer, evaluate on iTeach-HumanPlay, and run MSMFormer as a live ROS node.
3	🦾 SceneReplica (external)	Grasping and pick-and-place: iTeach-UOIS segmentation → Contact-GraspNet → GTO. How to run →
🔧	📍 iTeach (this repo) · src/hololens_utils	Still used by the current system: HoloDevicePortal.py uploads ROSConnectionConfig.json to the HoloLens, plus the hl2ss streaming helpers. The network setup is documented in iTeachSkillsApp.

Caution

Vital HoloLens step: before each new setup, upload ROSConnectionConfig.json (with RosIPAddress set to your robot) to the HoloLens LocalAppData → iTechDemo_… → LocalState folder through the Windows Device Portal web tool. Otherwise the app cannot reach ROS. Step-by-step guide →

Tip

Setting up the whole system? Start with the system diagram and five-terminal setup in iTeachSkillsApp. It covers the ROS topics, the voice commands, and how to point the HoloLens at your robot with ROSConnectionConfig.json.

📦 Data & Checkpoints

_{All hosted on UTD Box.}

	D5 · 5 controlled scenes	D40 · 40 scenes	Test · 902 samples
iTeach-HumanPlay dataset	Download	Download	Download
Fine-tuned MSMFormer	Download	Download	–

🛠️ Hardware

_{Everything the system needs is in this frame: a Fetch carrying an RGB-D camera and an RTX 4090 laptop that runs inference, SAM2 and fine-tuning onboard, and a human wearing a HoloLens 2 for the live overlay and gaze + voice annotation.}

🤖 Robot	Fetch mobile manipulator with a head RGB-D camera
💻 Compute	Laptop with an RTX 4090 for onboard inference and fine-tuning
🥽 Mixed reality	Microsoft HoloLens 2
🎮 Teleoperation	PS4 controller
🌐 Network	Wired Ethernet + Wi-Fi hotspot

🗒️ Lab setup reference — the exact working configuration (environment exports, ROS and network config, and a software-in-the-loop test harness that replays a recorded scene) is archived in docs/handoff/. Addresses there are placeholders; substitute your own.

_{⬆ back to contents}

---

🗄️ Legacy (archived): iTeach v1 · Door & Handle Detection

Warning

Archived. No longer maintained. Everything below, and the src/ (except src/hololens_utils/, which the current system still uses), toolkit/, dataloader/, hololens_app/ and hf_demo/ folders, covers the earlier version of iTeach. That version is a Mixed Reality labelling loop for door and handle detection with a YOLOv5-based model (DH-YOLO), the IRVLUTD DoorHandle dataset and the iTeachLabeller HoloLens app. It is kept only for reference and reproducibility. For the current system, see Code.

_{iTeach v1: Mixed Reality labelling loop for door and handle detection.}

🚀 Getting started in 3 steps

1 · Install the iTeachLabeller app on the HoloLens 2   📱 Build it from hololens_app/  ·  🛠️ Install video

2 · Set up the laptop and robot   📚 Follow src/README.md

3 · Teach   🤖 Drive the robot, collect failure samples, label them and fine-tune.  ·  🎬 Real-world demo

🎦 A walkthrough of the hardware, network and scripts is in this video. Detailed steps are in its description.

📁 Directory structure

Each directory has its own README.

Folder	Contents
🧪 src/	Main experiment files
🛠️ toolkit/	iTeach toolkit (DH-YOLO inference)
📱 hololens_app/	iTeachLabeller HoloLens app source
🗃️ dataloader/	PyTorch dataloader for the IRVLUTD DoorHandle dataset
🤗 hf_demo/	DH-YOLO Hugging Face space (git submodule: git submodule update --init hf_demo, needs git-lfs)

📦 Publishing toolkit / dataloader to PyPI

Run these with each new build:

rm -rf build/ dist/               # also remove the corresponding .egg-info directory
python setup.py sdist bdist_wheel # bump the version in setup.py first
twine upload dist/*               # needs your PyPI token

_{⬆ back to contents}

---

📜 License

Released under the MIT License, © 2024-2026 Intelligent Robotics and Vision Lab (IRVL), The University of Texas at Dallas.

_{Third-party code keeps its own license: the YOLOv5 code under src/iTeach/yolov5/ (and the YOLOv5-derived DH-YOLO code) is AGPL-3.0.}

📚 Citation

If iTeach helps your research, please cite:

@misc{padalunkal2024iteach,
      title={iTeach: In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception},
      author={Jishnu Jaykumar P and Cole Salvato and Vinaya Bomnale and Jikai Wang and Ayush Bhardwaj and Jin-Ryong Kim and Yu Xiang},
      year={2026},
      eprint={2410.09072},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2410.09072},
}

📬 Contact

💬 Questions & ideas	Discussion forum
🛠️ Bugs	Open an issue
📧 Direct	Jishnu

🙏 Acknowledgements

This work was supported by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005, the Sony Research Award Program, and the National Science Foundation (NSF) under Grant No. 2346528. We thank Sai Haneesh Allu for his assistance with the real-world experiments.

_{Built at the Intelligent Robotics and Vision Lab, The University of Texas at Dallas}