Robust Reinforcement Learning with Relevance Vector Machines

Minwoo Lee and Charles Anderson

Abstract:
Function approximation methods, such as neural networks, radial basis functions, and support vector machines, have been used in reinforcement learning to deal with large state spaces. However, they can become unstable with changes in the samples state distributions and require many samples for good estimations of value functions. Recently, Bayesian approaches to reinforcement learning have shown advantages in the exploration-exploitation tradeoff and in lower sampling costs. This paper proposes a novel reinforcement learning framework that uses the relevance vector machines (RVM) as a function approximator, which incrementally accumulates knowledge from experiences based on the sparseness of the RVM model. This gradual knowledge construction process increases the stability and robustness of reinforcement learning by preventing possible forgetting. In addition, RVM's low sampling costs improve the learning speed. The approach is examined in the popular benchmark problems of pole-balancing and mountain car.