In cooperative multi-agent reinforcement learning (MARL), efficient communication among agents requires the reduction of excessive message exchange at run-time to make it practical for real-world applications. This paper proposes a novel communication scheme, Attention-based Multi-Agent Cooperation (AMAC), that reduces overhead and shared information by exchanging only relevant messages across agents to coordinate decision-making and improve load balancing in networks. Experiments show that AMAC can significantly lower inter-agent communications overhead and learning complexity at the network controller level without degrading performance. The results demonstrate that our method actually outperforms multiple MARL benchmarks in Key Performance Indicators KPIs (such as throughput, delay, jitter), and Key Quality Indicators KQIs (such as QoE, average video bitrate, stalling).