LES is used to study self-excited transverse modes in an atmospheric, square combustor (p = 1 bar). Simulations over a range of different mass flow rates show that transverse modes are present for all cases and exhibit varying RMS pressure amplitudes up to 0.4 bar. Analysis of LES results shows that transverse modes are due to a lock-in mechanism between an hydrodynamic unstable mode typical of swirling flows (the PVC mode or Processing Vortex Core) and the cavity modes. A control method using damping compliant walls (named DCWs) is applied to control the acoustic mode in the LES and to characterize the PVC in the absence of acoustic forcing. This method shows that the highest pressure oscillations appear when the PVC frequency is close to the frequency of the first transverse acoustic mode. A 3D Helmholtz solver is then used to predict the stability limits obtained by 3D LES. To capture transverse modes, a new flame transfer function (FTF) formulation is derived where local heat release perturbations are controlled by the orthoradial acoustic velocity fluctuations. The FTF is measured in the LES and when it is included in the Helmholtz solver, it allows to recover the stability zones observed in the LES.