CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Karan Ahuja, Vivian Shen, Cathy Fang, Nathan Riopelle, Andy Kong, Chris Harrison

PDFpaperLinkpublication

Abstract

We present a new and practical method for capturing user body pose in virtual reality experiences: integrating cameras into handheld controllers, where batteries, computation and wireless communication already exist. By virtue of the hands operating in front of the user during many VR interactions, our controller-borne cameras can capture a superior view of the body for digitization. Our pipeline composites multiple camera views together, performs 3D body pose estimation, uses this data to control a rigged human model with inverse kinematics, and exposes the resulting user avatar to end user applications. We developed a series of demo applications illustrating the potential of our approach and more leg-centric interactions, such as balancing games and kicking soccer balls. We describe our proof-of-concept hardware and software, as well as results from our user study, which point to imminent feasibility.

teaser

Media

Citation

Ahuja, K., Shen, V., Fang, C. M., Riopelle, N., Kong, A., & Harrison, C. (2022, April). Controllerpose: inside-out body capture with VR controller cameras. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-13).

BibTeX

@inproceedings{ahuja2022controllerpose,
title={Controllerpose: inside-out body capture with VR controller cameras},
author={Ahuja, Karan and Shen, Vivian and Fang, Cathy Mengying and Riopelle, Nathan and Kong, Andy and Harrison, Chris},
booktitle={Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems},
pages={1--13},
year={2022}
}