We consider three-tier network architecture modeled with two physical nodes in tandem where an autonomous agent controls the number of active resources on each node. We analyse the learning of auto-scaling strategies in order to optimise both performance and energy consumption of the whole system. We compare several model-based reinforcement learning with model-free Q-learning algorithm. The relevance of these algorithms is to faster update Q-value function with an additional planning phase allowed by approximated model of the dynamics of the environment. Secondly, we consider the same tandem queue scenario with MMPP (Markov modulated Poisson process) for arrivals. In this context, the arrival rate is varying over time and this information is hidden to the agent. Our goal is to assess the robustness of such model-based reinforcement learning algorithms in this particular scenario.