The shear layer instability in the flow between two counter-rotating disks enclosed by a cylinder is investigated experimentally and numerically, for radius-to-height ratio Γ = R/h between 2 and 21. For sufficiently large rotation ratio, the internal shear layer that separates two regions of opposite azimuthal velocities is prone to an azimuthal symmetry breaking, which is investigated experimentally by means of visualization and particle image velocimetry. The associated pattern is a combination of a sharp-cornered polygonal pattern, as observed by Lopez et al. (2002) for low aspect ratio, surrounded by a set of spiral arms, first described by Gauthier et al. (2002) for high aspect ratio. The spiral arms result from the interaction of the shear layer instability with the Ekman boundary layer over the faster rotating disk. Stability curves and critical modes are experimentally measured for the whole range of aspect ratios, and are found to compare well with numerical simulations of the three-dimensional time-dependent Navier-Stokes equations over an extensive range of parameters. Measurements of a local Reynolds number based on the shear layer thickness confirm that a shear layer instability, with only weak curvature effect, is responsible for the observed patterns. This scenario is supported by the observed onset modes, which scale as the shear layer radius, and by the measured phase velocities.