In this study, we reexamine a recent optimal control simulation targeting the preparation of a superposition of two excited electronic states in the UV range in a complex molecular system. We revisit this control from the perspective of reinforcement learning, offering an efficient alternative to conventional quantum control methods. The two excited states are addressable by orthogonal polarizations and their superposition corresponds to a right or left localization of the electronic density. The pulse duration spans tens of femtoseconds to prevent excitation of higher excited bright states what leads to a strong perturbation by the nuclear motions. We modify an open source software by L. Giannelli et al., Phys. Lett. A, 434, 128054 (2022) that implements reinforcement learning with Lindblad dynamics, to introduce non-Markovianity of the surrounding either by timedependent rates or more exactly by using the hierarchical equations of motion with the QuTiP-BoFiN package. This extension opens the way to wider applications for non-Markovian environments, in particular when the active system interacts with a highly structured noise.