Boosted Bellman Residual Minimization Handling Expert Demonstrations

Abstract : This paper addresses the problem of batch Reinforcement Learning with Expert Demonstrations (RLED). In RLED, the goal is to find an optimal policy of a Markov Decision Process (MDP), using a data set of fixed sampled transitions of the MDP as well as a data set of fixed expert demonstrations. This is slightly different from the batch Reinforcement Learning (RL) framework where only fixed sampled transitions of the MDP are available. Thus, the aim of this article is to propose algorithms that leverage those expert data. The idea proposed here differs from the Approximate Dynamic Programming methods in the sense that we minimize the Optimal Bellman Residual (OBR), where the minimization is guided by constraints defined by the expert demonstrations. This choice is motivated by the the fact that controlling the OBR implies controlling the distance between the estimated and optimal quality functions. However, this method presents some difficulties as the criterion to minimize is non-convex, non-differentiable and biased. Those difficulties are overcome via the embedding of distributions in a Reproducing Kernel Hilbert Space (RKHS) and a boosting technique which allows obtaining non-parametric algorithms. Finally, our algorithms are compared to the only state of the art algorithm, Approximate Policy Iteration with Demonstrations (APID) algorithm, in different experimental settings.
Document type :
Conference papers
Complete list of metadatas

Cited literature [25 references]  Display  Hide  Download

https://hal-supelec.archives-ouvertes.fr/hal-01060953
Contributor : Sébastien van Luchene <>
Submitted on : Thursday, September 4, 2014 - 4:21:08 PM
Last modification on : Wednesday, July 31, 2019 - 4:18:03 PM
Long-term archiving on : Friday, December 5, 2014 - 10:39:44 AM

File

supelec885.pdf
Files produced by the author(s)

Identifiers

Citation

Bilal Piot, Matthieu Geist, Olivier Pietquin. Boosted Bellman Residual Minimization Handling Expert Demonstrations. European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. pp.549-564, ⟨10.1007/978-3-662-44851-9_35⟩. ⟨hal-01060953⟩

Share

Metrics

Record views

814

Files downloads

613