HAMEX - a Handwritten and Audio Dataset of Mathematical Expressions

Solen Quiniou (1) , Harold Mouchère (2) , Sebastian Peña Saldarriaga (3) , Christian Viard-Gaudin (2) , Emmanuel Morin (1) , Simon Petitrenaud (4) , Sofiane Medjkoune (2)

1 LINA - Laboratoire d'Informatique de Nantes Atlantique
2 IRCCyN - Institut de Recherche en Communications et en Cybernétique de Nantes
3 Synchromedia
4 LIUM - Laboratoire d'Informatique de l'Université du Mans

In this paper, we present HAMEX, a new public dataset that contains mathematical expressions available in their on-line handwritten form and in their audio spoken form. We have designed this dataset so that, given a mathematical expression, its handwritten signal and its audio signal can be used jointly to design multimodal recognition systems. Here, we describe the different steps that allowed us to acquire this dataset, from the creation of the mathematical expression corpora (including expressions from Wikipedia pages) to the segmentation and the transcription of the collected data, via the data collection process itself. Currently, the dataset contains 4 350 on-line handwritten mathematical expressions written by 58 writers, and the corresponding audio expressions (in French) spoken by 58 speakers. The ground truth is also provided both for the handwritten expressions (as INKML files with the digital ink, the symbol segmentation, and the MATHML structure) and for the audio expressions (as XML files with the transcriptions of the spoken expressions).

Keywords: dataset; mathematical expressions; handwriting recognition; speech recognition; multimodality

Domains: Signal and Image Processing; Signal and Image processing; Computer Vision and Pattern Recognition [cs.CV]

Solen Quiniou, Harold Mouchère, Sebastian Peña Saldarriaga, Christian Viard-Gaudin, Emmanuel Morin, et al.. HAMEX - a Handwritten and Audio Dataset of Mathematical Expressions. 11th International Conference on Document Analysis and Recognition, ICDAR 2011, Sep 2011, Beijing, China. ⟨hal-00615210⟩