This work presents the analysis of a transverse combustion instability in a reduced-scale rocket engine. The study is conducted on a time-resolved database of three-dimensional fields obtained via large-eddy simulation. The physical mechanisms involved in the response of the coaxial hydrogen/oxygen flames are discussed through the analysis of the Rayleigh term in the disturbance-energy equation. The interaction between acoustics and vorticity, also explicit in the disturbance-energy balance, is shown to be the main damping mechanism for this instability. The relative contributions of Rayleigh and damping terms, depending on the position of the flame with respect to the acoustic field, are discussed. The results give new insight into the phenomenology of transverse combustion instabilities. Finally, the applicability of spectral analysis on the nonlinear Rayleigh and dissipation terms is discussed.