Detached shear layers in a rotating container, known as Stewartson layers, become unstable for a critical shear measured by the Rossby number Ro. To study rapidly rotating flows at asymptotically small Rossby and Ekman numbers E, a quasigeostrophic (QG) model is developed, whose main original feature is to enforce mass conservation properly, and valid for any container provided that the top and bottom have finite slopes. Asymptotic scalings of the Stewartson instability are deduced from the linear QG model, extending the previous analyses to large slopes (as in a sphere). For a flat container, the critical Rossby number evolves as E^(3/4) and the instability can be understood as a shear instability. For a depth-varying container, the instability is quite different: it is a Rossby wave with a critical Rossby number proportional to beta E^(1/2), where beta is related to the slope. A numerical implementation of the QG model is used to determine the critical parameters for different Ekman numbers (as low as 10-10) and to describe the instability at the onset for different geometries. We also investigate the asymmetry between positive and negative Ro and propose an explanation based on the location of the instability. For flat cylindrical containers, the QG numerical results are directly compared to existing experimental data obtained by Niino and Misawa [J. Atmos. Sci. 41, 1992 (1984)] and Fruh and Read [J. Fluid Mech. 383, 143 (1999)]. We present new experimental results of the destabilization of a Stewartson layer in a rotating spheroid, caused by the differential rotation of two disks or a central inner core. These experiments are compared to the numerical QG results obtained in a split sphere, validating the QG model, and allowing us to show its limits. For spherical shells, the experimental critical curves agree to the ones obtained by three-dimensional calculations of Hollerbach [J. Fluid Mech. 492, 289 (2003)].