Second order nonlinearities in whispering gallery mode resonators are highly investigated for their many applications such as wavelength converters, entangled photon sources and generation of frequency combs. In such systems, depending on the material under scrutiny, the derivation of quasi-phase matching equations can lead to the appearance of additional quanta in the selection rule on the azimuthal confinement order. Here, we demonstrate that these additional quanta show up due to the Berry phase experienced by the transverse spin angular momentum components of the whispering gallery modes during their circulation within the resonator. We first detail the case of Zinc-blende materials and then generalize this theory to other crystal symmetries relevant for integrated photonics. A calculation app is available on github.com/Alex-l-r/QPM_BerryPhase_WGM