Over-optimism in bioinformatics: an illustration - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Bioinformatics Année : 2010

Over-optimism in bioinformatics: an illustration

Résumé

In statistical bioinformatics research, different optimization mechanisms potentially lead to “over-optimism” in published papers. The present empirical study illustrates these mechanisms through a concrete example from an active research field. The investigated sources of over-optimism include the optimization of the data sets, of the settings, of the competing methods and, most importantly, of the method's characteristics. We consider a “promising” new classification algorithm that turns out to yield disappointing results in terms of error rate, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. We quantitatively demonstrate that this disappointing method can artificially seem superior to existing approaches if we “fish for significance”. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should be validated using “fresh” validation data sets. The R codes and preprocessed versions of the data sets as well as additional files can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020−professuren/boulesteix/overoptimism/,such that the study is completely reproducible.
Fichier principal
Vignette du fichier
Jelizarow2010.pdf (180.62 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00514107 , version 1 (01-09-2010)

Identifiants

  • HAL Id : hal-00514107 , version 1

Citer

Monika Jelizarow, Vincent Guillemot, Arthur Tenenhaus, K. Strimmer, Anne-Laure Boulesteix. Over-optimism in bioinformatics: an illustration. Bioinformatics, 2010, 26, pp.1990-1998. ⟨hal-00514107⟩
112 Consultations
305 Téléchargements

Partager

Gmail Facebook X LinkedIn More