A survey on metrics for the evaluation of user simulations

Abstract : User simulation is an important research area in the field of spoken dialogue systems (SDSs) because collecting and annotating real human-machine interactions is often expensive and time-consuming. However, such data are generally required for designing, training and assessing dialogue systems. User simulations are especially needed when using machine learning methods for optimizing dialogue management strategies such as Reinforcement Learning, where the amount of data necessary for training is larger than existing corpora. The quality of the user simulation is therefore of crucial importance because it dramatically influences the results in terms of SDS performance analysis and the learnt strategy. Assessment of the quality of simulated dialogues and user simulation methods is an open issue and, although assessment metrics are required, there is no commonly adopted metric. In this paper, we give a survey of User Simulations Metrics in the literature, propose some extensions and discuss these metrics in terms of a list of desired features.
Document type :
Journal articles
Complete list of metadatas

https://hal-supelec.archives-ouvertes.fr/hal-00771654
Contributor : Sébastien van Luchene <>
Submitted on : Wednesday, January 9, 2013 - 10:28:50 AM
Last modification on : Wednesday, July 31, 2019 - 4:18:03 PM

Identifiers

Collections

Citation

Olivier Pietquin, Helen Hastie. A survey on metrics for the evaluation of user simulations. Knowledge Engineering Review, Cambridge University Press (CUP), 2013, 28 (1), pp.59-73. ⟨10.1017/S0269888912000343⟩. ⟨hal-00771654⟩

Share

Metrics

Record views

116