CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
AwardFast Company Innovation by Design Finalist

Pose-on-the-Go: Approximating Partial User Pose with Smartphone Sensor Fusion and Inverse Kinematics

Karan Ahuja, Sven Mayer, Mayank Goel, Chris Harrison

PDFpaperLinkpublicationgithubcode

Abstract

We present Pose-on-the-Go, a full-body pose estimation system that uses sensors already found in today’s smartphones. This stands in contrast to prior systems, which require worn or external sensors. We achieve this result via extensive sensor fusion, leveraging a phone’s front and rear cameras, the user-facing depth camera, touchscreen, and IMU. Even still, we are missing data about a user’s body (e.g., angle of the elbow joint), and so we use inverse kinematics to estimate and animate probable body poses. We provide a detailed evaluation of our system, benchmarking it against a professional-grade Vicon tracking system. We conclude with a series of demonstration applications that underscore the unique potential of our approach, which could be enabled on many modern smartphones with a simple software update.

APP

Check out our APP in Apple store

Media

Citation

Ahuja, K., Mayer, S., Goel, M., & Harrison, C. (2021, May). Pose-on-the-go: Approximating user pose with smartphone sensor fusion and inverse kinematics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-12).

BibTeX

@inproceedings{ahuja2021pose,
title={Pose-on-the-go: Approximating user pose with smartphone sensor fusion and inverse kinematics},
author={Ahuja, Karan and Mayer, Sven and Goel, Mayank and Harrison, Chris},
booktitle={Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},
pages={1--12},
year={2021}
}