Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search

Zahy Bnaya; Alon Palombo; Rami Puzis; Ariel Felner

doi:10.1609/socs.v6i1.18378

Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search

Zahy Bnaya, Alon Palombo, Rami Puzis, Ariel Felner

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states’ values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.

Original language	English
Title of host publication	Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015
Editors	Levi Lelis, Roni Stern
Publisher	Association for the Advancement of Artificial Intelligence
Pages	156-160
Number of pages	5
ISBN (Electronic)	9781577357322
DOIs	https://doi.org/10.1609/socs.v6i1.18378
State	Published - 2015
Externally published	Yes
Event	8th Annual Symposium on Combinatorial Search, SoCS 2015 - Ein Gedi, Israel Duration: 11 Jun 2015 → 13 Jun 2015

Publication series

Name	Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015
Volume	2015-January

Conference

Conference	8th Annual Symposium on Combinatorial Search, SoCS 2015
Country/Territory	Israel
City	Ein Gedi
Period	11/06/15 → 13/06/15

Bibliographical note

Publisher Copyright:
Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Funding

The research was supported by Israel Science Foundation (ISF) under grant #417/13 to Ariel Felner.

Funders	Funder number
Israel Science Foundation	417/13

Access to Document

10.1609/socs.v6i1.18378

Cite this

Bnaya, Z., Palombo, A., Puzis, R., & Felner, A. (2015). Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search. In L. Lelis, & R. Stern (Eds.), Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015 (pp. 156-160). (Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015; Vol. 2015-January). Association for the Advancement of Artificial Intelligence. https://doi.org/10.1609/socs.v6i1.18378

Bnaya, Zahy ; Palombo, Alon ; Puzis, Rami et al. / Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search. Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015. editor / Levi Lelis ; Roni Stern. Association for the Advancement of Artificial Intelligence, 2015. pp. 156-160 (Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015).

@inproceedings{ab2d524a48944f07be5845b56515077f,

title = "Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search",

abstract = "Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states{\textquoteright} values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.",

author = "Zahy Bnaya and Alon Palombo and Rami Puzis and Ariel Felner",

note = "Publisher Copyright: Copyright {\textcopyright} 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.; 8th Annual Symposium on Combinatorial Search, SoCS 2015 ; Conference date: 11-06-2015 Through 13-06-2015",

year = "2015",

doi = "10.1609/socs.v6i1.18378",

language = "אנגלית",

series = "Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015",

publisher = "Association for the Advancement of Artificial Intelligence",

pages = "156--160",

editor = "Levi Lelis and Roni Stern",

booktitle = "Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015",

}

Bnaya, Z, Palombo, A, Puzis, R & Felner, A 2015, Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search. in L Lelis & R Stern (eds), Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015. Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015, vol. 2015-January, Association for the Advancement of Artificial Intelligence, pp. 156-160, 8th Annual Symposium on Combinatorial Search, SoCS 2015, Ein Gedi, Israel, 11/06/15. https://doi.org/10.1609/socs.v6i1.18378

Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search. / Bnaya, Zahy; Palombo, Alon; Puzis, Rami et al.
Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015. ed. / Levi Lelis; Roni Stern. Association for the Advancement of Artificial Intelligence, 2015. p. 156-160 (Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015; Vol. 2015-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search

AU - Bnaya, Zahy

AU - Palombo, Alon

AU - Puzis, Rami

AU - Felner, Ariel

PY - 2015

Y1 - 2015

N2 - Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states’ values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.

AB - Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states’ values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.

UR - http://www.scopus.com/inward/record.url?scp=85048669921&partnerID=8YFLogxK

U2 - 10.1609/socs.v6i1.18378

DO - 10.1609/socs.v6i1.18378

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85048669921

T3 - Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015

SP - 156

EP - 160

BT - Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015

A2 - Lelis, Levi

A2 - Stern, Roni

PB - Association for the Advancement of Artificial Intelligence

T2 - 8th Annual Symposium on Combinatorial Search, SoCS 2015

Y2 - 11 June 2015 through 13 June 2015

ER -

Bnaya Z, Palombo A, Puzis R, Felner A. Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search. In Lelis L, Stern R, editors, Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015. Association for the Advancement of Artificial Intelligence. 2015. p. 156-160. (Proceedings of the 8th Annual Symposium on Combinatorial Search, SoCS 2015). doi: 10.1609/socs.v6i1.18378

Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this