Executive Summary
Transferring human motion onto robots in real time remains one of the most practically demanding problems in industrial robotics. Existing pipelines either regress body pose from single RGB frames, which produce inherently noisy and scale-ambiguous outputs, or rely on proprietary motion-capture systems that cannot generalise across robots. The Humano Retargeting Framework addresses both constraints with a modular architecture that grounds human skeletal motion in metric 3D space via RGB-D sensing, filters it temporally with an Extended Kalman Filter, converts it into scale-invariant joint angles, and retargets those angles onto any robot defined by a URDF file via a task-space inverse kinematics model. The pipeline ships with three simulation backends (Isaac Lab, MuJoCo and ROS 2/Gazebo) and a real-hardware interface, all selectable via a single YAML configuration file. A robot operator can switch from Unitree G1 to Fourier GR1 to AgiBot X2 by changing one line. The safety filter layer, with velocity clamping, torque estimation and dead-man hold-last, ensures that signal loss and tracking failure never cause uncontrolled motion on a physical platform.
System Specifications
[1] The pipeline starts with multi-view RGB-D captures of a skilled worker performing tasks with hardware-synchronised frames. Camera intrinsics, extrinsics and per-frame metadata (exposure, gain and wall-clock timestamp) are logged alongside every frame. Humans working in the scene are detected, and their ViT features are extracted along with confidence-weighted bounding boxes. [2] Global body orientation, 24 joint rotations in axis-angle form, 10 shape coefficients (betas) and camera-frame translation are estimated. These parameters are passed to a statistical body model that produces surface mesh vertices and canonical joint positions. [3] Raw depth-projected joints are noisy at the 1-5 cm level due to depth-sensor noise and model-fitting residuals. An Extended Kalman Filter is maintained per joint with a constant-velocity motion model to produce a temporally coherent set of 24 joint positions. [4] Before retargeting, the filtered 3D joint positions are converted to human joint angles. For each joint in the kinematic chain, the angle is computed from the dot product of the parent bone vector and child bone vector. The retargeter is initialised from the robot URDF file, and the IK solver solves multi-task optimisation at each frame. Proportional scaling maps human segment lengths to robot segment lengths before task targets are set. [5] The safety filter sits between the retargeter and the robot, with velocity limits to prevent violent single- frame jumps from reaching the robot, torque estimation to approximate expected joint torque, and a dead-man switch to avoid haphazard behaviour.
