Project Logo

EgoHTR

Egocentric 4D Demonstrations of Human Terrain Traversal

Alex Brandes 1 Haig Conti Georges Sajelian4
Manthan Patel1 Dominik Hollidt 1 Chenhao Li 1 Matthias Heyrman 1 Oliver Hausdörfer 4 Manuel Kaufmann 1 Xi Wang 1 Jonas Frey 2,3 Angela P. Schoellig 4 Christian Holz 1 Marc Pollefeys 1 Marco Hutter 1
1 ETH Zurich 2 Stanford 3 UC Berkeley 4 TU Munich
arXiv Paper Video Dataset (coming soon) Code (coming soon)

Abstract

Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between humanoid learning and scene reconstruction, we introduce the Egocentric Human-Terrain Reconstruction (EgoHTR) dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of egocentric wearables and a portable 3D scanner. The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build foundational, context-aware robots that reliably traverse uneven terrain.

Dataset Explorer

Select an explorer tab below to view the statistics, available modalities and scene episodes.

7
Scenes
55
Sequences
1.37h
Total Duration
~150k
Frames @ 30fps
8
Subjects (4F / 4M)
36
Multi-View Seq. (0.88h)
0.7h
Mocap GT Test Subset

Select a tab to visualize the correspoding modality of an example episode.

Methodology

Capture system overview
# Sensor Role Placement Required
1Project Aria Glasses (Gen. 1)Egocentric captureHead-mountedRequired
2Rokoko Pro IIBody motion captureBody-worn (IMU suit)Required
3Leica BLK2GOScene reconstructionHandheld scannerRequired
4Project Aria Glasses (2nd)Multi-view / exocentric captureObserver-wornOptional
5Fixed-View CameraThird-person reference viewStaticOptional

The scene-aware retargeting pipeline used for the G1 embodiement builds on the work of OmniRetarget, GMR and CoACD.

Application

We train perceptive whole-body control policies from reconstructed reference motions and deploy them on a Unitree G1 humanoid.

We use our dataset to benchmark state-of-the-art reconstruction and estimation pipelines.

Capture system overview
Capture system overview
Capture system overview

BibTeX

@misc{brandes2026egohtregocentric4ddemonstrations,
      title={EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal}, 
      author={Alex Brandes and Haig Conti Georges Sajelian and Manthan Patel and Dominik Hollidt and Chenhao Li and Matthias Heyrman and Oliver Hausdoerfer and Manuel Kaufmann and Xi Wang and Jonas Frey and Angela P. Schoellig and Christian Holz and Marc Pollefeys and Marco Hutter},
      year={2026},
      eprint={2607.13472},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.13472}, 
}