Abstract
In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in advance that there are two types of hidden states; one is common to all arms and evolves according to a Markovian distribution, and the other is unique to each arm and is distributed according to an i.i.d. process that is unique to each arm. We present an algorithm and regret analysis to this problem. Surprisingly, we can recover the hidden states and maintain logarithmic regret in the case of a convex polytope action set. Furthermore, we show that the structural side information leads to expected regret that does not depend on the number of extreme points in the action space. Therefore, we obtain practical solutions even in high dimensional problems.
| Original language | English |
|---|---|
| Article number | 9335534 |
| Pages (from-to) | 1108-1123 |
| Number of pages | 16 |
| Journal | IEEE Transactions on Signal Processing |
| Volume | 69 |
| DOIs | |
| State | Published - 2021 |
Bibliographical note
Publisher Copyright:© 1991-2012 IEEE.
Funding
Manuscript received June 2, 2020; revised December 1, 2020; accepted January 13, 2021. Date of publication January 25, 2021; date of current version February 12, 2021. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Victor Elvira. The work of Amir Leshem was supported by ISF Grant 1644/18. (Corresponding author: Michal Yemini.) Michal Yemini is with the Department of Electrical Engineering, Stanford University, Stanford CA 94305 USA (e-mail: [email protected]).
| Funders | Funder number |
|---|---|
| Israel Science Foundation | 1644/18 |
Keywords
- Linear bandit
- Markov decision process
- hidden states
- restless bandit
- side information