In this paper, we address the linear stability analysis of a confined potential vortex with a free surface. This particular flow has been recently used by Tophøj et al. (Phys. Rev. Lett., vol. 110(19), 2013, article 194502) as a model for the swirling flow of fluid in an open cylindrical container, driven by rotating the bottom plate (the rotating bottom experiment) to explain the so-called rotating polygons instability (Vatistas J. Fluid Mech., vol. 217, 1990, pp. 241–248; Jansson et al., Phys. Rev. Lett., vol. 96, 2006, article 174502) in terms of surface wave interactions leading to resonance. Global linear stability results are complemented by a Wentzel–Kramers–Brillouin–Jeffreys (WKBJ) analysis in the shallow-water limit as well as new experimental observations. It is found that global stability results predict additional resonances that cannot be captured by the simple wave coupling model presented in Tophøj et al. (2013). Both the main resonances (thought to be at the root of the rotating polygons) and these secondary resonances are interpreted in terms of over-reflection phenomena by the WKBJ analysis. Finally, we provide experimental evidence for a secondary resonance supporting the numerical and theoretical analysis presented. These different methods and observations allow to support the unstable wave coupling mechanism as the physical process at the origin of the polygonal patterns observed in free-surface rotating flows.