Researchers at The George Washington University are developing an innovative device and method that can quickly and easily register and learn an user’s personalized hand gestures in real time. The invention pertains to depth sensors for VR headsets with hand tracking, and the software algorithms to learn (register) user’s individualized hand gestures and recognize (classify) the gestures in VR headsets in real time. The invented device and method provides for a real-time training (registration) and recognition (classification) framework for hands-free and natural interaction methods in virtual reality VR environments.
Many virtual reality (VR) systems utilize hand-held controllers to track user’s hand motion and to get button commands. Although controllers support the interface to be more accurate and explicit, it is inconvenient for users to keep holding the controllers during interaction. Hand gesture communication is one of the definite solutions to reduce the inconvenience and maintain the explicit command transmission. However, in the case of hand gesture recognition, it is difficult to use a pre-trained gesture recognition model because each user has different customized gestures. Moreover, in order to additionally train the recognizer with extra user data, the processes of data acquisition and management are tiresome. It also takes a long time to train the model and it does not guarantee that the data is well reflected in the existing model. Previous methods (e.g. Hololens built-in system) do not allow user registration or are not real-time training, and the number of gestures that can be recognized is limited. Furthermore, conventional systems allow only one depth sensor per computer.
The method and system of this invention utilizes multiple depth sensors to track hands in wider motion range. Thereby the invented method provides a framework that can quickly and easily learn user’s personalized hand gesture in real time. This technology is a powerful innovation in Virtual Reality, with significant commercial potential in a growing number of environments.
Applications:
1. Virtual Reality
2. Depth sensors for VR headsets with hand tracking, and software algorithms to learn user’s individualized hand gestures and recognize (classify) the gestures in VR headsets in real time
Advantages:
1. Easily and quickly register and learn an user’s personalized hand gestures in real time
2. Tracks a wide range of hand motions
