VR application gives users ‘superhuman’ ability to open and control tools

By employing machine learning, the team developed ‘HotGestures’, an innovative approach analogous to hot keys in desktop applications, allowing users to control a range of 3D modelling tools through simple hand movements.

‘HotGestures’ enables users to create figures and shapes in virtual reality without the need to navigate traditional menus, maintaining an uninterrupted focus on their tasks. This system not only simplifies the interaction but also enhances the user experience by reducing the cognitive load typically associated with tool selection in virtual environments.

Professor Per Ola Kristensson, from Cambridge’s Department of Engineering and co-Director of the Centre for Human-Inspired Artificial Intelligence, highlighted the significance of this advancement: “Users gain some qualities when using VR, but very few people want to use it for an extended period of time.” Beyond ergonomic and visual fatigue challenges, Kristensson pointed out that VR needs to offer unique capabilities, distinct from those in the real world.

Moving beyond the concept of hot keys, the Cambridge team’s ‘HotGestures’ allow users to open and control tools in VR through intuitive gestures. For instance, a cutting motion activates the scissor tool, while a spray motion brings up the spray can tool. This intuitive system allows users to seamlessly switch between tools without pausing their workflow.

“We all communicate using our hands in the real world, so it made sense to extend this form of communication to the virtual world,” said Kristensson, emphasising the natural integration of HotGestures into the VR experience.

The researchers developed a neural network gesture recognition system capable of distinguishing ten different gestures associated with building 3D models. This system ensures accuracy and prevents false activations, distinguishing between intentional commands and normal hand movements. In trials, HotGestures proved to be faster, more distinctive, and easier to use compared to conventional menu-based interactions.

“There is no VR system currently available that can do this,” Kristensson stated. By transcending traditional interaction methods like keyboards and mice, the team aims to provide VR users with almost superhuman capabilities. Kristensson envisions this technology becoming a standard for VR interactions, moving away from outdated metaphors and towards more intuitive methods. “We need new ways of interacting with technology, and we think this is a step in that direction. When done right, VR can be like magic,” he added.

The source code and dataset have been made publicly available, encouraging VR application designers to integrate this innovative system into their products. This research, partially supported by the Engineering and Physical Sciences Research Council (EPSRC), marks a significant leap forward in the field of virtual reality, paving the way for more intuitive, efficient, and engaging user interactions.