Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors 论文

详细信息

发表日期

2017-01-01

发表年份

2017

摘要

We present an algorithm for fusing multi-viewpoint video (MVV) with inertial measurement
\nunit (IMU) sensor data to accurately estimate 3D human pose. A 3-D convolutional
\nneural network is used to learn a pose embedding from volumetric probabilistic
\nvisual hull data (PVH) derived from the MVV frames. We incorporate this model within
\na dual stream network integrating pose embeddings derived from MVV and a forward
\nkinematic solve of the IMU data. A temporal model (LSTM) is incorporated within
\nboth streams prior to their fusion. Hybrid pose inference using these two complementary
\ndata sources is shown to resolve ambiguities within each sensor modality, yielding improved
\naccuracy over prior methods. A further contribution of this work is a new hybrid
\nMVV dataset (TotalCapture) comprising video, IMU and a skeletal joint ground truth
\nderived from a commercial motion capture system. The dataset is available online at
\nhttp://cvssp.org/data/totalcapture/.