🤖 HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation

Sai Haneesh Allu* · Jishnu Jaykumar P* · Ninad Khargonkar · Tyler Summers · Jian Yao · Yu Xiang
_{* Equal Contribution}

TL;DR — A robot watches a human demonstration once (captured from the robot's point of view through an AR headset) and then repeats the same mobile manipulation task in new environments, even when objects are placed differently.

---

📑 Table of Contents

• 📝 Abstract
• 🧩 Pipeline at a Glance
• ⚙️ Setup
  • 1️⃣ Clone the Repository and set environment variables
  • 2️⃣ Set up each stage
  • 3️⃣ Keep submodules up to date
• 🙏 Acknowledgements
• 📚 Citation

---

📝 Abstract

We introduce a novel system for human-to-robot trajectory transfer that enables robots to manipulate objects by learning from human demonstration videos. The system consists of four modules. The first module is a data collection module that is designed to collect human demonstration videos from the point of view of a robot using an AR headset. The second module is a video understanding module that detects objects and extracts 3D human-hand trajectories from demonstration videos. The third module transfers a human-hand trajectory into a reference trajectory of a robot end-effector in 3D space. The last module utilizes a trajectory optimization algorithm to solve a trajectory in the robot configuration space that can follow the end-effector trajectory transferred from the human demonstration. Consequently, these modules enable a robot to watch a human demonstration video once and then repeat the same mobile manipulation task in different environments, even when objects are placed differently from the demonstrations.

---

🧩 Pipeline at a Glance

Stage	Module	What it does	Docs
I	👓 Data Capture	HoloLens 2 app + robot-side ROS endpoint to record human demos from the robot's viewpoint	dc/
II	🎬 Video Information Extraction (VIE)	Object detection & tracking (GroundingDINO + SAM 2) and 3D hand-trajectory extraction (HaMeR)	vie/
III	✋➡️🦾 Grasp Transfer	Maps human-hand poses to Fetch gripper poses; BundleSDF for object pose estimation at execution time	vie/
IV	🚀 Trajectory Tracking Optimization (TTO)	Solves for robot joint trajectories that follow the transferred end-effector path — in simulation and the real world	tto/

flowchart LR
    A["👓 Stage I<br/>Data Capture<br/>(HoloLens 2)"] --> B["🎬 Stage II<br/>Video Information<br/>Extraction"]
    B --> C["✋➡️🦾 Stage III<br/>Hand-to-Gripper<br/>Transfer"]
    C --> D["🚀 Stage IV<br/>Trajectory Tracking<br/>Optimization"]
    D --> E["🤖 Task Execution<br/>(Fetch Robot)"]

Loading

---

⚙️ Setup

1️⃣ Clone the Repository and set environment variables

Clone the repository recursively to include all submodules:

git clone --recursive https://github.com/IRVLUTD/HRT1 && cd HRT1
# Create a conda environment from scratch
conda create -n hrt1 python=3.10  # Python 3.10 required for samv2 and hamer dependencies
conda activate hrt1

# Set your CUDA_HOME environment variable
export CUDA_HOME=/usr/local/cuda

2️⃣ Set up each stage

This codebase, built on top of the robokit and gto tools. Refer Readme document for each of the below utilities to setup the pipeline, in this order:

1. 👓 Stage I → dc/
2. 🎬 Stage II & III → vie/ (also contains the BundleSDF module used for object pose estimation during execution)
3. 🚀 Stage IV → tto/

See 🧩 Pipeline at a Glance for what each stage does.

3️⃣ Keep submodules up to date

To get the latest changes from the submodules

git submodule sync
git submodule update --remote --recursive

---

🙏 Acknowledgements

HRT1 builds on several excellent open-source projects, including robokit, GroundingDINO, SAM 2, HaMeR, BundleSDF, and ROS-TCP-Endpoint. We thank their authors for making their work available.

---

📚 Citation

Please cite this work if it helps in your research

@misc{2025hrt1,
  title={HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation}, 
  author={Sai Haneesh Allu* and Jishnu Jaykumar P* and Ninad Khargonkar and Tyler Summers and Jian Yao and Yu Xiang},
  year={2025},
  url={https://arxiv.org/abs/2510.21026}, 
}

---

Released under the MIT License