We propose a policy improvement algorithm for Reinforcement Learning (RL) termed Rerouted Behavior Improvement (RBI). RBI is designed to take into account the evaluation errors of the Q-function. Such errors are common in RL when learning the Qvalue from finite experience data. Greedy policies or even constrained policy optimization algorithms that ignore these errors may suffer from an improvement penalty (i.e., a policy impairment). To reduce the penalty, the idea of RBI is to attenuate rapid policy changes to actions that were rarely sampled. This approach is shown to avoid catastrophic performance degradation and reduce regret when learning from a batch of transition samples. Through a two-armed bandit example, we show that it also increases data efficiency when the optimal action has a high variance. We evaluate RBI in two tasks in the Atari Learning Environment: (1) learning from observations of multiple behavior policies and (2) iterative RL. Our results demonstrate the advantage of RBI over greedy policies and other constrained policy optimization algorithms both in learning from observations and in RL tasks.
|Title of host publication||Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020|
|Publisher||International Joint Conferences on Artificial Intelligence|
|Number of pages||9|
|State||Published - 2020|
|Event||29th International Joint Conference on Artificial Intelligence, IJCAI 2020 - Yokohama, Japan|
Duration: 1 Jan 2021 → …
|Name||IJCAI International Joint Conference on Artificial Intelligence|
|Conference||29th International Joint Conference on Artificial Intelligence, IJCAI 2020|
|Period||1/01/21 → …|
Bibliographical noteFunding Information:
∗This work was supported by the Israel Innovation Authority.
© 2020 Inst. Sci. inf., Univ. Defence in Belgrade. All rights reserved.