The free surface flow in a cylindrical tank over a rotating bottom is known to support spectacular three-dimensional patterns, including deformation of the inner free surface into the shape of rotating polygons, and sloshing behavior of the upper free surface (e.g. Iga et al., FDR 2014, same issue). Through a stability analysis of a simplified model of this flow, we show that such patterns can be explained as a resonance mechanism involving different families of waves. The approach extends a previous work (Tophoj et al., PRL 2013) which explained the rotating polygons as an interaction between gravity waves and centrifugal waves, under the assumption that the base flow can be modeled as a potential vortex. We show that this previous model is justified for strong rotation rates (Dry-Potential case), and that for weaker rotations it can be improved by introducing an inner vortex core in solid-body rotation, which either extends to the center of the plate (Wet case) or surrounds a dry central region (Dry-Composite case). The study of this improved model predicts two new kind of instabilities. The first occurs at low rotations (Wet case) and results from an interaction between gravity waves and the Kelvin-Kirchhoff wave (namely, oscillation of the boundary of the vortex core). This instability is proposed to be at the origin of the sloshing phenomenon. The second new instability occurs, for moderate rotations, (Dry-Composite case) as an interaction between gravity waves and a "Kelvin-Centrifugal" wave characterized by deformation of the inner surface and the vortex core boundary in opposite directions. This instability exists for all azimuthal wave numbers starting from m = 1, this case corresponding to a "monogon" pattern.