Exploring multimodality micro-gesture detection for emotion understanding

Abstract

This study explores multimodality micro-gesture detection and emotion understanding on the identity-free video dataset for Micro-gesture Understanding and Emotion analysis (iMiGUE). Micro-gestures identified as subtle unintentional gestures, can be associated with the emotional state of an individual. The specific objectives of this study include studying videos of players from a post-match interview in the iMiGUE dataset to identify key-frames where micro-gestures occur and provide a detailed description for emotion understanding. Furthermore, this study explores the multimodal form of representation for micro-gesture detection and emotion recognition and also compares the performances of unimodal and multimodal frameworks. Through the use of action video detection models and multimodal large language models (MLLMs), micro-gestures are detected and combined to form a textual description in each video for emotion understanding using large language models. Two major experiments involve micro-gesture detection using MLLMs and micro-gesture description using MLLMs. The end goal of both experiments is to correctly classify the emotional state of a player as positive or negative. The experiments provide accuracies of 53% and 55% respectively on emotion recognition level. These results are compared to past studies that utilized unimodal frameworks to observe that this study provides performances comparable to past studies. In conclusion, the multimodality framework proposed in this study can be utilized for micro-gesture detection and emotion understanding.