Content for  TR 22.837  Word version:  19.3.0

As a new way of human-device interface, gesture recognition enables a more intuitive interaction between humans and machines, compared to the conventional text or GUI-based interfaces. Common applications of gesture recognition include touchless control of mobile devices, such as smartphones, laptops, and smart watches. Compared to other use cases, such as sports monitoring or sleep monitoring, gesture recognition requires higher resolution, higher update rate, and lower latency, which makes it more challenging in terms of resource utilization and processing complexity. Gesture recognition identifies motions and postures of human body parts, such as head, hands, and fingers. As shown in Figure 5.29.1-1, gesture recognition can be applied to various applications such as human motion recognition, keystroke detection, sign language recognition and touchless control.

In this use case, we focus on application of gesture recognition for touchless control and immersive (i.e. XR) application. For touchless control, the identified gestures are then interpreted to specific behaviours or operations of the device, including locking/unlocking a screen, increasing/decreasing volume, and navigating forward/backward web pages. For XR application, the position tracking and mapping of the human body is a basic requirement that permeates XR application to provide the immersive experience [55]. Hereby a variety of sensors are integrated in the XR devices to measure the movement of the human body (e.g., head, eye, hand, arm, etc) in order to simulate normal human mimicry. Besides, the hand tracking goes beyond the simulation and enables a natural and intuitive way of the interaction between the human and the machine compared with the use of the physical controller. The gesture recognition and hand tracking becomes a vital function in XR applications, especially when the human and the controlled object stay in physical and virtual world separately. For both touchless control and XR application, NR-based RF sensing is suitable for gesture recognition because certain RF signals can detect coarse body movements and the RF signals are not susceptible to the ambient illumination condition and occlusions in the environment. In addition, NR-based RF sensing allows for coarse hand tracking in a lower-complexity and economic manner.

There are two roommates, Jose and Bob, both of whom subscribed to MNO A, which has deployed RAN entity (e.g., an indoor base station) supporting NR-based sensing. Jose subscribes to the touchless user interface service and his mobile device has NR sensing capability. Bob subscribes to the immersive interaction service, provided by both MNO A and XR application (e.g. game, sports training) provider AppX on the access rights of the interaction information relevant to Bob's hands. Bob's UEs (e.g. smartphone, XR device such as headset) are capable of NR-based sensing and Bob's XR device also have other sensors (e.g. IMU) embedded on the device.

Due to the RF sensing capability in Jose's mobile device and a nearby RAN entity, Jose's gestures are detected and used to navigate the social media posts on his phone. Similarly, due to the RF sensing capability in Bob's smartphone, XR device and a nearby RAN entity, Bob's coarse gesture and the motion of his hands will be recognized and tracked correctly. The avatar in XR application will show the correct gesture and execute the correct action triggered by the gesture.

None.

[PR 5.29.6-1]