Distributionally robust chance-constrained Markov decision processes with random payoff
Résumé
A Markov Decision Process (MDP) is a framework used for decision-making. In an MDP problem, the decision maker's goal is to maximize the expected discounted value of future rewards while navigating through different states controlled by a Markov chain. In this paper, we focus on the case where the transition probabilities vector is deterministic, while the reward vector is uncertain and follow a partially known distribution. We employ a distributionally robust chance constraints approach to model the MDP. This approach entails the construction of potential distributions of reward vector, characterized by moments or statistical metrics. We explore two situations for these uncertainty sets: one where the reward vector has a real support and another where it is constrained to be nonnegative. In the case of a real support, we demonstrate that solving the distributionally robust chance-constrained Markov decision process is mathematically equivalent to a second-order cone programming problem for moments and ϕ-divergence uncertainty sets. For Wasserstein distance uncertainty sets, it becomes a mixed-integer second-order cone programming problem. In contrast, when dealing with nonnegative reward vector, the equivalent optimization problems are different. Moments-based uncertainty sets lead to a copositive optimization problem, while Wasserstein distance-based uncertainty sets result in a biconvex optimization problem. To illustrate the practical application of these 1 methods, we examine a machine replacement problem and present results conducted on randomly generated instances to showcase the effectiveness of our proposed methods.
Mots clés
Markov decision process Distributionally robust chance constraints Second-order cone programming Copositive optimization Mix-integer second-order cone programming Biconvex optimization
Markov decision process
Distributionally robust chance constraints
Second-order cone programming
Copositive optimization
Mix-integer second-order cone programming
Biconvex optimization
Fichier principal
DRO_MDP_Random_payoff__Applied_Maths_and_Optimization_template_.pdf (612.77 Ko)
Télécharger le fichier
Origine | Fichiers produits par l'(les) auteur(s) |
---|