Executive Summary
Creating photorealistic and physics-aware personalized avatars that can be driven in real time has remained a significant challenge in digital human research. Existing pipelines either require expensive hardware setups, manual post-processing or sacrifice body-shape personalisation for runtime performance. Humano avatar pipeline addresses these constraints by delivering a fully automated workflow that transforms dual-perspective views of a Pan-Tilt-Zoom-Depth camera session into a live, textured, body-shape-personalised avatar driven in real time inside Blender, Isaac Sim and MuJoCo.
The pipeline starts with an optimizer-driven avatar creation phase that captures, fits, and textures the avatar and skeleton, followed by a live skeleton-tracking phase that runs continuously, streaming vertex and bone updates from a trained diffusion-based pose tracking model into physics engines at approximately 30 Hz.
System Specifications
Humano Avatar Pipeline is positioned as a modular digital-human infrastructure stack with a calibrated capture system, a whole-body perception layer, an expressive mesh reconstruction backend, a metric world-grounding module, a photorealistic appearance builder, and a simulation/robotics bridge.
[1] The capture module treats our PTZD camera as a calibrated sensor and not only as a video source. For each session, the pipeline exploits the camera intrinsics, native resolution, stream codec, exposure/gain settings, PTZ preset, frame time and wall-clock time. This metadata becomes essential for mesh recovery into metric world coordinates in the third stage. [2] Perceptual modelling uses a layered perception front-end with a detector, tracker, whole-body pose estimator, segmentation and quality estimator. The detector produces person regions, the tracker maintains IDs, the whole-body model estimates body, hand, face and foot keypoints, and SAM 2-style segmentation produces masks for person and clothing regions. The quality estimator decides whether to trust, smooth or drop a frame. [3] The core human representation remains SMPLer-X style because it unifies body shape, pose, articulated hands and facial expression in a mesh topology compatible with UV transfer and downstream rigging. [4] The system fuses camera calibration, monocular metric depth, pelvis/torso landmarks and temporal filtering with stereo depth serving as the depth backbone depending on speed, quality and intrinsics requirements. [5] The UV projection step extends from a single texture pass into a multi-view atlas optimization. For every camera frame and visible mesh triangle, source pixels are sampled into the UV atlas using view-angle weighting, segmentation masks and confidence from the mesh/depth fit. The atlas stores color, confidence, source frame ID and optionally normal/roughness estimates. [6] The live system structures pose packets rather than unversioned raw arrays. Each packet contains schema_version, track_id, timestamp_ns, sequence_id, coordinate_frame, root_pose, joint_rotations, joint_positions, vertices_optional, confidence and quality_flags.
