Communication Dans Un Congrès Année : 2025

A Simple Integration of Epistemic Logic and Reinforcement Learning

Résumé

We propose an integration of epistemic logic with reinforcement learning via a semantics that uses the concept of belief bases. In our framework, an agent's subjective state is identified with their belief base, which captures the agent's personal representation of the environment. The agent's subjective state is distinguished from the global state, which captures the overall information about the environment and about the agent's belief base from an external perspective. We instantiate the concepts of global state and subjective state in Partially-Observable Markov Decision Process (POMDPs), defining so-called Belief Base POMDPs (BB-POMDPs). We show that in our epistemic framework, we can use the beliefs of the learning agent to formalize and implement a natural form of shielding, which prevents agents from performing actions that are not known to be safe. Our implementation of shielding relies on a model-checking algorithm to automatically verify whether a given fact is deducible from the agent's belief base.

We perform a case study of model-free reinforcement learning on a simple wumpus scenario, using a variant of Q-learning on the agent's subjective states, using the agent's beliefs for reward shaping and shielding. In particular, our experiments show that our version of shielding can successfully protect the agent from harm while improving the utility of the learned policy.

Fichier principal
Vignette du fichier
p686.pdf (1.02 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-05109660 , version 1 (12-06-2025)

Licence

Identifiants

  • HAL Id : hal-05109660 , version 1

Citer

Thorsten Engesser, Thibaut Le Marre, Emiliano Lorini, François Schwarzentruber, Bruno Zanuttini. A Simple Integration of Epistemic Logic and Reinforcement Learning. Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), May 2025, Detroit (Michigan), United States. pp.686-694. ⟨hal-05109660⟩
1464 Consultations
328 Téléchargements

Partager

  • More