This paper proposes a modular deep reinforcement learning framework integrated with digital twin technology for optimizing the control of Reconfigurable Manufacturing Systems (RMS). The framework employs hierarchical deep reinforcement learning agents for scheduling and reconfiguration decisions across decentralized digital twins of individual Reconfigurable Machine Tools (RMT). The digital twins enable real-time monitoring, simulation, and visualization to inform the reinforcement learning agents. The modular architecture aligns with RMS goals of adaptability and rapid reconfiguration. The reconfiguration agent selects optimal machine configurations based on assessments of job queues, tardiness costs, and due dates. The scheduling agent optimizes job sequencing given the current configuration. The RMS environment coordinates the machine agents for overall optimization. Predictive maintenance capabilities are also incorporated within the digital twins. This integration of digital twins and deep reinforcement learning provides capabilities for optimal control of resilient and efficient RMS, to be responsive to dynamic manufacturing environments.