Developing efficient load-balancing techniques remains a persistent research challenge as modern WiFi networks evolve into increasingly complex environments, incorporating new enhancements in their standards. For instance, DQL-MultiMDP is a load-balancing algorithm that learns an optimal STA-to-AP association policy to ensure user fairness and optimize network performance in dense and dynamic WiFi networks. The algorithm leverages a Multi-Markov Decision Process (Multi-MDP) strategy to accommodate the fluctuating number of devices caused by their switching on/off. However, scalability challenges arise due to the exponential expansion of the action space. In this paper, we propose a divide-and-conquer approach that extends the algorithm to operate in extremely large deployments: a dynamic partitioning mechanism divides the network into clusters and assigns a sub-controller to manage the STA-to-AP association in each cluster independently, and a coordination mechanism enables them to exchange their training updates. Experimental investigations validate the effectiveness of the approach and motivate future work.