Deep Learning Algorithms for Multimodal Interaction Using Speech and Motion Data in Virtual Reality Systems

Authors :

Ahmed Zubair and Fatima Al Rashed

Address :

Faculty of Computer science, American University of Sharjah, UAE

Abstract :

Multimodal interaction (MMI) represented by speech and motion data (SMD) has enormous potential in virtual reality (VR) systems. However, real-time synchronization, context-sensitive interpretation, and effective fusion of heterogeneous data modalities remain open. The study presents a deep learning-based framework that fuses speech and motion data to provide better performance in interaction. This study proposes a novel method called MMI-CNNRNN that combines a Convolutional Neural Network (CNN) that features extraction in speech with a Recurrent Neural Network (RNN) for temporal motion analysis, integrated into a Transformer-based architecture to enhance the contextual understanding and responsiveness of the system. In this regard, the performance of the proposed framework is evaluated using benchmark multimodal datasets such as the IEMOCAP dataset. These results represent a 20% increase in interaction accuracy and a 15% latency reduction compared to unimodal and early fusion methods. The fusion of CNN and RNN mechanisms translates into more natural and intuitive interactions, making both the assistive device and the VR environment more adaptive and user-friendly. Concluding from the findings of the proposed work, efficient multimodal system development supports better accessibility and engagement among users with various needs.

1.Introduction

Technological advances have brought into view the possibility of developing new multimodal interaction systems that can link both the physical and digital worlds. It is now possible to render virtual content in real time based on the agent's context data, thanks to network and computer power improvements that allow several devices to connect to the Internet with sufficient performance [1]. Researchers in human-machine interaction (HMI) seek to improve human and machine collaboration to enable machines to adjust to the needs of an application [2]. The ability to coordinate the use of one's speech with several hand gestures at the same time is an increasingly hot subject. This coordination is exceedingly complex, operating on multiple levels and timescales [3]. Voice and gestures both have their benefits as multimodal and unimodal inputs. It is necessary to perform a thorough assessment of these strengths [4]. Everyday speech and gestures, when combined, can convey more nuanced meaning than either modality could on its own. We still need to learn more about these modalities to draw any firm conclusions about how they work together or independently in AR/HMD settings [5]. To keep the human-robot collaboration (HRC) efficient performance going, it still needs to be easier for robots to respond to human actions in manufacturing and achieve natural, accurate, and real-time identification [6].

The gaming, healthcare, and training industries are just a few of the many that have benefited from the recent developments in virtual reality (VR) technology. There is a rising focus on developing adaptive systems that can intelligently adjust to user states in real-time as VR advances [7]. Virtual reality settings naturally offer a multi-sensory experience, satisfying various tastes using sight, sound, and, in some cases, touch [8]. There is a notable knowledge vacuum in this area because, although previous research has investigated VLM applications in many domains, such as the problematic jobs of surgical aid and traffic hazard prediction, virtual reality has ignored it chiefly [9].

By combining speech and motion data (SMD), these systems can create more immersive, adaptive, and user-friendly interfaces. However, significant challenges remain in achieving real-time synchronization, context-sensitive interpretation, and compelling fusion of heterogeneous data modalities. Existing unimodal approaches or simple, early fusion techniques often need to catch up in handling the complexity and dynamism of multimodal inputs, limiting their applicability in real-world VR systems. This paper addresses these challenges by proposing a new deep learning-based framework, called MMI-CNNRNN, which is specifically designed for fusing speech and motion data. It leverages a CNN for extracting robust features from speech inputs and an RNN for capturing the temporal dynamics of motion data; these are then integrated in a Transformer-based architecture, enhancing contextual understanding and making the system responsive to complex multimodal interactions.

The key significance of this study is,

• To develop an innovative fusion framework, MMI-CNNRNN integrates CNNs for speech feature extraction, RNNs for temporal motion analysis, and Transformers for enhanced contextual understanding in multimodal interaction systems.
• To achieve significant performance improvements, including a 20% increase in interaction accuracy and a 15% reduction in latency, validated through benchmark datasets like IEMOCAP.
• To demonstrate the practical impact of the proposed framework by enhancing accessibility and user engagement in VR environments, mainly supporting diverse user needs through efficient multimodal system design.

2.Literature Survey

Park, K. B. et al. [10] presented a concept of hands-free HRI using eye gazing and head motion-based multimodal gestures with deep learning-based object detection in the MR environment to develop improved task efficiency with reduced error in noisy conditions that could emerge from conventional manipulation. Results have shown how the proposed method allows for fast and efficient object manipulation, higher task completion times, and better performance concerning cutting-edge methodologies despite issues related to marker tracking stability.

Kang T. et al. [11] proposed a hand interface for intuitive interactions in immersive VR, mapping real-world hand gestures to virtual actions without needing a GUI using a Convolutional Neural Network (CNN). This interface is proposed to improve user experiences, enhance immersion, and offer an affordable and realistic interaction structure. The results showed improved satisfaction, ease of use, and presence compared to traditional GUI methods, which user surveys and statistical analysis verified.

Gupta S. et al. [12] presented a multimodal engagement detection system, incorporating DL models such as VGG-19 and ResNet-50, by tracking facial expressions, eye movements, and head position to assess student engagement during an e-learning session instantly. It will be proposed to improve student engagement and receive immediate feedback so that educators may adjust their methods according to the engagement level. It yielded 92.58% accuracy in detecting engagement, effectively encouraging a more interactive and responsive online learning environment.

Ahmad Z. et al. [13] suggested a deep fusion model that integrates many modalities, merging spectrograms with 1D electrocardiogram signals that are both raw and processed., to estimate three distinct stress levels in a Virtual Reality environment. It is essential beyond simple binary stress classification that would allow more engaging biofeedback applications. Results demonstrated a 9% increase in accuracy from state-of-the-art machine learning models and a 2.5% increase from baseline deep learning models. Therefore, this proves the model's efficacy in real-time stress estimation during immersive experiences.

A deep learning model called Multi-Input CNN-LSTM was suggested by Masuda, N., & Yairi, I. E. [14] for the purpose of fear level classification using multimodal peripheral physiological inputs and multichannel EEG. The model used a combination of CNNs for feature extraction and LSTMs for sequence learning, doing away with the need for human attribute selection. Aimed at enhancing fear detection for mental health applications, the model achieved impressive results, classifying four fear levels with 98.79% accuracy and a 99.01% F1 score in 10-fold cross-validation, outperforming previous methods.

Ravva, P. U. et al. [15] presented a two-step machine learning framework that predicts the intention of upper limb motion during the performance of VR tasks for rehabilitation. It uses a neural network for segment prediction and LSTM models to obtain direction movement by integrating gaze data and resistance measurements obtained through wearable sensors. It can enhance rehabilitation outcomes by allowing precise, real-time assessment of motion intentions, which is impossible with conventional therapy methods. It achieved high accuracy, with 96.72% for diamond tasks and 97.44% for circle tasks, proving its effectiveness.

3. Proposed Methodology

The proposed framework MMI-CNNRNN jointly captures speech and motion information harmoniously to improve the MMI in VR systems. First, speech signals and motion data are acquired by using sensors and devices integrated with VR systems. These inputs are pre-processed to reduce noise and normalize them in format. A CNN extracts speech features, discovering intricate patterns in the audio, while an RNN processes the motion data for temporal dynamics analysis. These features are fused, after feature extraction, into a transformer-based architecture that enhances contextual understanding by aligning speech and motion. This creates more natural and responsive interactions within virtual environments, thus making it more accessible and increasing user engagement. The framework demonstrates significant performance gains, proving an effective solution for intuitive VR applications. Figure 1 illustrates the procedure of the suggested approach.

a. Data Acquistion

Speech Data: Speech captured through microphones or VR headsets provides raw audio signals that are converted into waveforms or spectrograms. These representations highlight some key acoustic features that can be deeply analyzed and used to extract features for understanding speech patterns, thereby improving real-time interactions in virtual reality systems.

Motion Data: Motion sensors, such as IMUs, Kinect cameras, and VR controllers, generate time-series data for a user's motion. These data inherently contain temporal and spatial dynamics, allowing for accurate motion analysis and smooth interaction in VR environments.

b. Preprocessing (Speech Data)

Spectrogram Generation Using Short-Time Fourier Transform (STFT): One of the most important preprocessing steps in audio signal analysis is transforming audio data into spectrograms. A spectrogram is a time-frequency representation of the signal and allows the extraction of features in both time and frequency domains. The Short-Time Fourier Transform (STFT) is applied to an audio signal x(t), expressed mathematically as in equation 1.

where w(n-t) is the window function (Hamming window) ω is the frequency domain. STFT chops the signal into overlapping frames; for every segment, it applies the window function and then calculates the Fourier Transform for the signal within the window. This yields a time-frequency matrix that can be used to build spectrograms.

Computation of Mel-Frequency Cepstral Coefficients (MFCCs): MFCCs are some of the most used features in speech-related audio feature extractions due to their effectiveness in approximating the nonlinear frequency perception of the human auditory system. MFCCs result from transforming the log power spectrum into the Mel frequency scale of the audio signal. Formulation of the k -th order MFCC is performed as in equation 2.

where X_n represents the magnitude spectrum obtained from the STFT, N denotes the number of Mel filters applied and k represents the coefficient index. This process involves transforming the frequency spectrum into a Mel scale, performing a logarithmic transform, and performing a Discrete Cosine Transform (DCT) that decorates the filter bank energies.

Data Normalization: Normalization is one of the important steps that helps enhance the audio's robustness, improves noise, and diminishes variability among the samples. Normalization normalizes the audio signal x(t), by using its mean μ _x and standard deviation σ_( x), which can be mathematically represented as in equation 3.

This transformation will have zero mean and unit variance, which is quite helpful for convergence in training machine learning models and allows the generalization performance for an audio processing model

Preprocessing (Motion Data): Motion Data Preprocessing for Temporal Analysis: The preprocessing of time-series motion data involves critical steps such as noise filtering and normalization to ensure accuracy in temporal analysis. The Kalman Filter is applied to reduce sensor noise, which estimates the system’s state through two primary steps: forecasting and revision. In the prediction step, the estimated state x_(k∣k-1 )^' is determined using the equation 4, the error covariance P_(k|k-1) is calculated as in equation 5, the Kalman gain K_k is computed to minimize the estimation error is obtained as in equation 6 and The estimated state x_(k|k)^' is then refined using the measurement z_k is shown in equation 7.

Here,x_k^' represents the estimated state, P_k denotes the error covariance and K_k is the Kalman gain. The matrices A,B and H represent system dynamics while Q and R are the process and measurement noise covariances, respectively.

Normalization follows noise filtering to bring motion data, such as joint angles or coordinates, to a uniform scale. The normalized motion data M_norm (t) is computed as in equation 8.

where M(t) is the raw motion data, μ_M is the mean, and σ_M is the standard deviation. Equation 8 normalizes all the raw values of motion data to zero mean and unit variance, making various datasets more consistent and improving the successive analytical model's performance.

c. Speech Path Processing Using Convolutional Neural Networks (CNN)

The preprocessed speech data will be fed into CNN, which transforms the audio signal into visual or numerical forms, capturing frequency and amplitude over time, thus allowing for a rich feature set for analysis. The convolutional layers perform the actual convolution of the filter or kernel over the input data to extract important features. The output feature map F(i, j) at position (i, j) is computed as in equation 9.

I(m,n) indicates the input data, spectrogram or MFCC, K is the convolution kernel, also called a filter, and b is the bias term. The network can detect edges, frequencies, or other localized input features.

Pooling layers are used after convolution to reduce dimensionality while preserving important information. The most frequently used approach is max pooling, which down-samples the input by sliding a window of size f and taking the maximum value. Mathematically, the pooling output P(i,j) is expressed in equation 10.

It compresses the feature maps while preserving the most salient features, contributing to more efficient training and reducing overfitting. The resulting feature maps represent various aspects of speech patterns, including phonemes, prosody, and other acoustic characteristics, which are essential for speech recognition or classification tasks. Figure 2 shows the process of CNN in speech data.

Motion Path Processing Using Recurrent Neural Networks (RNN): The processing of time-series motion data, such as the joint angles or IMU sensor readings, through RNNs captures temporal patterns of movement and dynamics. In practice, LSTM is used to deal with such long-term dependencies; it keeps track of a cell state, c_t, together with several other gates that control information flow: an input gate, an forget gate, and an output gate. At each time step t, the input gate i_t, forget gate f_t, and output gate o_t are computed as in equation 11.

where x_t is the input at a time t, h_(t-1) is the hidden state from the previous time step, and σ represents the sigmoid activation function. These gates decide the flow of things to remember and forget, the output at every stage of the process. The cell state c_t is updated by combining the previous cell state c_(t-1) with the current input and forget gates:

Finally, the hidden state h_t, representing the output at the current time step, is calculated as in equation 13. Figure 3 shows the motion path process using RNN.

d. Fusion Layer with Transformer Integration

The presented work uses speech and motion data within multimodal systems, therefore including a fusion layer which merges the feature vectors extracted by CNNs for speech and RNNs for motion. The Transformer-based architecture exploits an attention mechanism to align and weigh on the importance of each modality dynamically. This mechanism helps in providing highlights on relevant features from CNN-extracted spectrogram or MFCC and RNN-encoded temporal motion representations on contextual relevance. The self-attention process will judge the mutual dependency of every speech and motion data point based on high values for information-rich features that are contextually more useful. This representation enhances the contextual insight of the system so that speech and motion inputs can be interpreted in an inter-preceding fashion. This kind of integration allows for higher-level decision-making, which is imperative in applications like assistive technologies and immersive virtual environments that rely on multimodal synchronization. Pseudocode 1 depicts the fusion layer integrated with a transformer.

		Pseudocode 1: Fusion Layer with Transformer Integration
Inputs: Speech and Motion Data 
# speech_features: Output feature vector from CNN (e.g.,spectrogram or MFCC) 
# motion_features: Output feature vector from RNN (e.g.,LSTM/GRU for motion data) 
# model_params: Transformer parameters (e.g.,number of attention heads,layers,etc.) 
function multimodal_fusion(speech_features,motion_features,model_params): 
    # Step 1: Project both feature vectors to the same dimension 
    speech_proj = LinearProjection(speech_features,model_params.hidden_dim)   
    motion_proj = LinearProjection(motion_features,model_params.hidden_dim ) 

    # Step 2: Combine features into a multimodal sequence 
    fused_sequence = concatenate([speech_proj,motion_proj],axis=sequence_length_axis)  
    # Step 3: Position encoding for temporal alignment 
    fused_sequence = add_position_encoding(fused_sequence) 
    # Step 4: Transformer Encoder with Attention Mechanism  
    for layer in range(model_params.num_layers): 
        # Multi-Head Self-Attention 
        attention_output = MultiHeadAttention(fused_sequence,fused_sequence,fused_sequence,
 model_params.num_heads) 
                # Add & Norm (Residual Connection + Layer Normalization) 
        fused_sequence = LayerNorm(fused_sequence + attention_output) 
                # Feed-Forward Network (FFN) 
        ffn_output = FeedForwardNetwork(fused_sequence,model_params.ffn_hidden_dim) 
                # Add & Norm (Residual Connection + Layer Normalization) 
        fused_sequence = LayerNorm(fused_sequence + ffn_output)   
    # Step 5: Extract the fused representation 
    fused_representation = Pooling(fused_sequence,pooling_type='global_average')   
        return fused_representation  # Final fused feature vector 
speech_input = extract_speech_features(raw_audio_data)  # Process through CNN 
motion_input = extract_motion_features(raw_motion_data)  # Process through RNN 
fused_output = multimodal_fusion(speech_input,motion_input,transformer_params) 
														

e. Multimodal Classifier

The classifier acts on the fused feature representations of speech and motion pathways for effective integration of information from both modalities, which ensures completeness in understanding the context of the interaction. It effectively allows the system to make sense of the complex input from the user by analyzing speech characteristics, such as tone or pitch, together with motion patterns related to gestures or movements. The classifier, through these complex signals from varied sources, ascertains subtleties- such as changes in speech tone or parallel gestures- and picks them up correctly for the overall responsiveness and accuracy of the interaction system.

Fused Feature Representation: Combines the output feature maps from the CNN (speech) and RNN (motion) pathways. This can be done through concatenation, attention mechanisms, or more advanced fusion techniques as in equation 14.

Classification Layer: These fused features are fed into one or many fully connected layers that eventually classify these features. These fully connected layers transform the combined information into an appropriate decision-making format. Each fully connected layer applies a linear transformation, where the fused feature vector. F_fused is first multiplied by a weight matrix W and added to a bias vector b to produce an intermediate output z . This can be represented as ( z = W∙ F_fused + b )Outputs are passed through activation functions to introduce non-linearity and allow the model to learn complex patterns. Common choices include ReLU for hidden layers, which helps the network learn intricate relationships, and Softmax for the final layer, which converts the outputs into probabilities for each class, thereby facilitating accurate classification.

Output Layer: The softmax activation function generates class probabilities in the output layer of the proposed multimodal system, which, in turn, allows for generating predictions about the user intent or specific actions like gestures or commands. Thus, the softmax function has been used in many multi-class classification frameworks. The softmax function determines the probability for each class by taking the exponential of the input zi corresponding to each i-th neuron in the output layer and normalizing it by the sum of exponentials of all N class inputs, as shown in equation 15.

Decision Making: During the decision-making phase of the proposed system, the final class prediction takes the class with the maximum probability from the softmax output. Mathematically, the process can be expressed as in equation 16.

where y^' denotes the forecasted class label, and z_i is the input of the i-th neuron in the output layer. The softmax function converts These inputs into a probability distribution over N classes. The class with the highest probability is chosen as the final prediction corresponding to a certain interaction command, emotional state, or gesture. This classification at the output allows the system to act accordingly and contextually, making correct and prompt decisions even in real-world applications.