Abstract
Reinforcement learning (RL) algorithms output policies specifying which action an agent should take in a given state. However, faults can sometimes arise during policy execution due to internal faults in the agent. As a result, actions may have unexpected effects. In this work, we aim to diagnose such faults and infer their root cause. We consider two types of diagnosis problems. In the first, which we call RLDXw, we assume we only know what a normal execution looks like. In the second, called RLDXs, we assume we have models for the faulty behavior of a component, which we call fault modes. The solution to RLDXw is a time step at which a fault occurred for the first time. The solution to RLDXs is more informative, represented as a fault mode according to which the RL task was executed. Solving those problems is useful in practice to facilitate efficient repair of faulty agents, since it can focus the repair efforts on specific actions. We formally define RLDXw and RLDXs and design two algorithms called WFMa and SFMa for solving them. We evaluate our algorithms on a benchmark of RL domains and discuss their strengths and limitations. When the number of the observed states increases, both WFMa and SFMa report a decrease in runtime (up to significantly 6.5 times faster). Additionally, the runtime of SFMa increases linearly with the increase in candidate fault modes.
Original language | English |
---|---|
Title of host publication | 35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024 |
Editors | Ingo Pill, Avraham Natan, Franz Wotawa |
Publisher | Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing |
ISBN (Electronic) | 9783959773560 |
DOIs | |
State | Published - 26 Nov 2024 |
Externally published | Yes |
Event | 35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024 - Vienna, Austria Duration: 4 Nov 2024 → 7 Nov 2024 |
Publication series
Name | OpenAccess Series in Informatics |
---|---|
Volume | 125 |
ISSN (Print) | 2190-6807 |
Conference
Conference | 35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024 |
---|---|
Country/Territory | Austria |
City | Vienna |
Period | 4/11/24 → 7/11/24 |
Bibliographical note
Publisher Copyright:© Avraham Natan, Roni Stern, and Meir Kalech.
Keywords
- Autonomous Systems
- Diagnosis
- Reinforcement Learning