Real-Time 3D Human Pose Reconstruction from Multi-Camera Views
Overview
This project reconstructs 3D human pose from live video streams captured by multiple cameras. For each frame, a YOLO pose model detects 2D keypoints in each view, and a triangulation step estimates the corresponding 3D joint coordinates in a shared world reference frame. The output is rendered as an interactive 3D skeleton in real time.
I built the full pipeline in Python, including camera geometry and triangulation logic, to keep direct control over each stage of the system. The implementation is designed to be flexible enough to switch between body and hand tracking, and to scale from two cameras to larger multi-camera setups.
Problem Context
A single camera view does not directly provide depth, because projecting a 3D scene onto a 2D image plane loses distance information. Multi-view geometry addresses that limitation. When the same keypoint is observed from two or more camera viewpoints with known poses, it becomes possible to estimate the original 3D location by triangulation.
This project applies that principle to human pose keypoints. Instead of reconstructing dense 3D geometry, it focuses on structured landmarks, which makes real-time operation practical while still capturing useful motion information.
Pipeline
1. Multi-Camera Capture
Each camera is defined in a shared coordinate system by position and orientation. In my setup, I used two Logitech BRIO 100 webcams placed on a desk and pointed toward the subject from different angles. Capture runs in separate threads, so frame acquisition from one camera does not block the others.
2. 2D Pose Estimation
Each incoming frame is processed by a YOLOv11 pose model. The system can run in two modes: body pose with 17 COCO keypoints, or hand pose with 21 landmarks. The detector outputs normalized 2D coordinates and confidence values for the detected keypoints.
3. 3D Triangulation
The reconstruction stage uses a pinhole camera model and homogeneous transforms. For each keypoint, the 2D observations from all available camera views are converted into a linear least-squares system. Solving that system gives the 3D point that best fits all observations.
The method is straightforward to scale: adding another camera simply adds additional constraints to the same solve step. In practice, this also helps robustness against noisy 2D detections.
4. Real-Time 3D Visualization
The reconstructed keypoints are displayed in a live Matplotlib 3D plot. Joint connections are drawn to form a readable skeleton, and body regions are color-coded to improve visual interpretation during motion.
Implementation Notes
The current implementation uses manually measured camera placement rather than a full checkerboard calibration workflow. This keeps setup simple and fast for experimentation, while still providing a workable geometric model for live reconstruction. The codebase also includes two runtime paths: a desktop-oriented version with interactive Matplotlib rendering, and a Jetson-targeted version where OpenCV handles visualization in a headless-friendly workflow.
Results
With two cameras, the system produces a stable live 3D skeleton for both body and hand tracking. End-to-end throughput is around 10 FPS in my tests, with most runtime spent on sequential YOLO inference. The triangulation solve itself is comparatively lightweight.
Current Limitations
The current version assumes cross-view correspondence implicitly by using the first detected person in each view. This works in single-subject scenes, but it is not sufficient for reliable multi-person reconstruction. In addition, the independent camera threads are not hardware-synchronized, so fast motions can show small temporal inconsistencies across views.
Accuracy also depends on camera baseline and placement quality. In practice, wider camera spacing improves depth conditioning, and small extrinsic measurement errors can be visible in the resulting 3D estimate.
Technology Stack
- Python 3.11+
- Ultralytics YOLOv11 for 2D pose estimation
- OpenCV for camera I/O and Jetson rendering path
- NumPy for transforms and least-squares solving
- Matplotlib for interactive 3D visualization
Future Work
The next steps are to add cross-view identity association for multi-person scenes, improve calibration with marker-based workflows, and reduce inference bottlenecks by batching camera inputs on GPU. I also plan to export reconstructed trajectories to standard motion formats for downstream analysis.