A Novel Method of Emotion Classification and Reconstruction Using VGGNet and StarGAN for Mixed-Reality Interactions in HKU Campusland Metaverse

Abstract

<p>Verbal, written, visual, and nonverbal communication are the four main forms of communication. Nonverbal communication, including facial expressions and body gestures, is an effective method for better understanding emotion in communication. With the recent growth of the metaverse, more people and companies started using the metaverse for their business activities. However, the current communication in the metaverse is mainly in written format. This means that the metaverse environment is unable to provide human-like interactions, preventing participants from joining the metaverse and having real-world social and business activities. In order to make human interactions in the metaverse more real, this research aims to develop a novel method for nonverbal communication in the metaverse. Therefore, we reviewed current methods of emotion classification and regeneration and proposed a novel method of emotion reconstruction in the metaverse, which includes the processes of facial emotion detection and reconstruction. We used the FER2013 dataset to train our improved model of VGG19-CNNs with a residual masking network and data augmentation for facial emotion classification. We compared our novel method with a baseline machine learning model, the support vector machine. Our novel model improved the accuracy from 50.2% of the baseline model to 70.8% of the VGG19 model and 88.33% of VGGNet with a residual masking network and data augmentation. Finally, we used the StarGAN model to reconstruct the detected emotion onto the avatar’s face as a case study in HKU Campusland.</p>