Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer

Etienne Labbé (1) , Julien Pinquier (1) , Thomas Pellegrini (1)

1 IRIT-SAMoVA - Équipe Structuration, Analyse et MOdélisation de documents Vidéo et Audio

In this work, we propose to study the performance of a model trained with a sentence embedding regression loss component for the Automated Audio Captioning task. This task aims to build systems that can describe audio content with a single sentence written in natural language. Most systems are trained with the standard Cross-Entropy loss, which does not take into account the semantic closeness of the sentence. We found that adding a sentence embedding loss term reduces overfitting, but also increased SPIDEr from 0.397 to 0.418 in our first setting on the AudioCaps corpus. When we increased the weight decay value, we found our model to be much closer to the current state-of-the-art scores, with a SPIDEr score up to 0.444 compared to a 0.475 score. Moreover, this model uses eight times less trainable parameters than the current state-of-the-art method Multi-TTA. In this training setting, the sentence embedding loss has no more impact on the model performance.

Keywords: sound event description; multitask learning; audio language task; overfitting; sentence embedding regression loss; semantic loss

Domains: Sound [cs.SD]

Etienne Labbé, Julien Pinquier, Thomas Pellegrini. Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer. 31st European Signal Processing Conference (EUSIPCO 2023), Sep 2023, Helsinki, Finland. ⟨10.48550/arXiv.2305.01482⟩. ⟨hal-04207519⟩