Realistic Transformation of Facial and Vocal Smiles in Real-Time Audiovisual Streams

Pablo Arias (1, 2, 3) , Catherine Soladie (4) , Oussema Bouafif (4) , Axel Roebel (1, 5, 2) , Renaud Seguier (4) , Jean-Julien Aucouturier (1, 2, 3)

1 STMS - Sciences et Technologies de la Musique et du Son
2 IRCAM - Institut de Recherche et Coordination Acoustique/Musique
3 STMS-PDS - Perception et design sonores
4 IETR - Institut d'Électronique et des Technologies du numéRique
5 Analyse et synthèse sonores [Paris]

Research in affective computing and cognitive science has shown the importance of emotional facial and vocal expressions during human-computer and human-human interactions. But, while models exist to control the display and interactive dynamics of emotional expressions, such as smiles, in embodied agents, these techniques can not be applied to video interactions between humans. In this work, we propose an audiovisual smile transformation algorithm able to manipulate an incoming video stream in real-time to parametrically control the amount of smile seen on the user's face and heard in their voice, while preserving other characteristics such as the user's identity or the timing and content of the interaction. The transformation is composed of separate audio and visual pipelines, both based on a warping technique informed by real-time detection of audio and visual landmarks. Taken together, these two parts constitute a unique audiovisual algorithm which, in addition to providing simultaneous real-time transformations of a real person's face and voice, allows to investigate the integration of both modalities of smiles in real-world social interactions.

Mots clés: Facial expressions; Vocal emotions; Video and audio signal processing; Real-time; Audiovisual; Index Terms-Smiling

Domaines: Son [cs.SD]; Traitement du signal et de l'image [eess.SP]; Neurosciences; Psychologie

Pablo Arias, Catherine Soladie, Oussema Bouafif, Axel Roebel, Renaud Seguier, et al.. Realistic Transformation of Facial and Vocal Smiles in Real-Time Audiovisual Streams. IEEE Transactions on Affective Computing, 2020, PP (99), pp.1-1. ⟨10.1109/TAFFC.2018.2811465⟩. ⟨hal-01712834⟩