Avatar Reaction to Multimodal Human Behavior

Baptiste Chopin (1) , Mohamed Daoudi (2, 3) , Angela Bartolo (4)

1 STARS - Spatio-Temporal Activity Recognition Systems
2 CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
3 IMT Nord Europe - Ecole nationale supérieure Mines-Télécom Lille Douai
4 SCALab - Laboratoire Sciences Cognitives et Sciences Affectives - UMR 9193

In this paper, we propose a virtual agent application. We develop a virtual agent that reacts to gestures and a virtual environment in which it can interact with the user. We capture motion with a Kinect V2 camera, predict the end of the motion and then classify it. The application also features a facial expression recognition module. In addition, to all these modules, we include also OpenAI conversation module. The application can also be used with a virtual reality headset.

Mots clés: avatar reaction; human motion prediction; facial expression

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Baptiste Chopin, Mohamed Daoudi, Angela Bartolo. Avatar Reaction to Multimodal Human Behavior. GHB 2023 - Generation of Human Face and Body Behavior Workshop in conjonction with ICIAP, Sep 2023, Udine, Italy. ⟨hal-04305332v2⟩