In modern-day manufacturing, it is imperative to react promptly to altering market requirements. Reconfigurable Manufacturing Systems (RMS) are a significant leap forward in achieving this criteria as they offer a flexible and affordable structure to comply with evolving production necessities. The ever-changing nature of RMS demands a sturdy induction of learning algorithms to persistently improve system configurations and scheduling. This study suggests that using Reinforcement Learning (RL), specifically, the Double Deep Q-Network (DDQN) algorithm, is a feasible way to navigate the intricate, multi-objective optimization landscape of RMS. Key points to consider regarding this study include cutting down tardiness costs, ensuring sustainability by reducing wasted liquid and gas emissions during production, optimizing makespan, and improving ergonomics by reducing operator intervention during system reconfiguration. Our proposal consists of two layers. Initially, we suggest a hierarchical and modular architecture for RMS which includes a multi-agent environment at the reconfigurable machine tool level, which improves agent interaction for optimal global results. Secondly, we incorporate DDQN to navigate the multi-objective space in a clever manner, resulting in more efficient and ergonomic reconfiguration and scheduling. The findings indicate that employing RL can help solve intricate optimization issues that come with contemporary manufacturing paradigms, clearing the path for Industry 5.0.