Boosted Bellman Residual Minimization Handling Expert Demonstrations

Abstract : This paper addresses the problem of batch Reinforcement Learning with Expert Demonstrations (RLED). In RLED, the goal is to find an optimal policy of a Markov Decision Process (MDP), using a data set of fixed sampled transitions of the MDP as well as a data set of fixed expert demonstrations. This is slightly different from the batch Reinforcement Learning (RL) framework where only fixed sampled transitions of the MDP are available. Thus, the aim of this article is to propose algorithms that leverage those expert data. The idea proposed here differs from the Approximate Dynamic Programming methods in the sense that we minimize the Optimal Bellman Residual (OBR), where the minimization is guided by constraints defined by the expert demonstrations. This choice is motivated by the the fact that controlling the OBR implies controlling the distance between the estimated and optimal quality functions. However, this method presents some difficulties as the criterion to minimize is non-convex, non-differentiable and biased. Those difficulties are overcome via the embedding of distributions in a Reproducing Kernel Hilbert Space (RKHS) and a boosting technique which allows obtaining non-parametric algorithms. Finally, our algorithms are compared to the only state of the art algorithm, Approximate Policy Iteration with Demonstrations (APID) algorithm, in different experimental settings.
Type de document :
Communication dans un congrès
European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. 8725, pp.549-564, 2014, Lecture Notes in Computer Science. 〈10.1007/978-3-662-44851-9_35〉
Liste complète des métadonnées

Littérature citée [25 références]  Voir  Masquer  Télécharger

https://hal-supelec.archives-ouvertes.fr/hal-01060953
Contributeur : Sébastien Van Luchene <>
Soumis le : jeudi 4 septembre 2014 - 16:21:08
Dernière modification le : mardi 24 avril 2018 - 13:37:40
Document(s) archivé(s) le : vendredi 5 décembre 2014 - 10:39:44

Fichier

supelec885.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Bilal Piot, Matthieu Geist, Olivier Pietquin. Boosted Bellman Residual Minimization Handling Expert Demonstrations. European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. 8725, pp.549-564, 2014, Lecture Notes in Computer Science. 〈10.1007/978-3-662-44851-9_35〉. 〈hal-01060953〉

Partager

Métriques

Consultations de la notice

544

Téléchargements de fichiers

292