Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search

Abstract : Local Policy Search is a popular reinforcement learning approach for handling large state spaces. Formally, it searches locally in a parameterized policy space in order to maximize the associated value function averaged over some pre-defined distribution. The best one can hope in general from such an approach is to get a local optimum of this criterion. The first contribution of this article is the following surprising result: if the policy space is convex, any (approximate) local optimum enjoys a global performance guarantee. Unfortunately, the convexity assumption is strong: it is not satisfied by commonly used parameterizations and designing a parameterization that induces this property seems hard. A natural so-lution to alleviate this issue consists in deriving an algorithm that solves the local policy search problem using a boosting approach (constrained to the convex hull of the policy space). The resulting algorithm turns out to be a slight generalization of conservative policy iteration; thus, our second contribution is to highlight an original connection between local policy search and approximate dynamic pro-gramming.
Document type :
Conference papers
Complete list of metadatas

Cited literature [25 references]  Display  Hide  Download

https://hal-supelec.archives-ouvertes.fr/hal-01086345
Contributor : Sébastien van Luchene <>
Submitted on : Monday, November 24, 2014 - 8:38:59 AM
Last modification on : Wednesday, July 31, 2019 - 4:18:02 PM
Long-term archiving on : Wednesday, February 25, 2015 - 10:15:51 AM

File

supelec886.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Bruno Scherrer, Matthieu Geist. Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search. ECMLPKDD 2014, Sep 2014, Nancy, France. pp.35 - 50, ⟨10.1007/978-3-662-44845-8_3⟩. ⟨hal-01086345⟩

Share

Metrics

Record views

179

Files downloads

153