This paper investigates the design of a challenging Game AI for a modern strategy game, which can be seen as a large-scale multiagent simulation of an historical military confrontation. As an alternative to the typical script-based approach used in industry, we test an approach where military units and leaders, organized in a hierarchy, learn to improve their collective behavior through playing repeated games. In order to allow the application of a reinforcement learning framework at each level of this complex hierarchical decision-making structure, we propose an abstraction mechanism that adapts semi-automatically the level of detail of the state and action representations to the level of the agent. We also study specifically various reward signals as well as inter-agent communication setups and show their impact on the Game AI performance, distinctively in offensive and defensive modes. The resulting Game AI achieves very good performance when compared with the existing commercial script-based solution.