The lateral shift of an optical beam undergoing Brillouin light scattering by a spin wave propagating along the interface between magnetic and dielectric media (Damon-Eshbach configuration) in the total internal reflection geometry is studied theoretically. Linear and quadratic magneto-optic terms in polarization are taken into account. It is shown that the lateral shift depends on the polarization (s- or p-) state of the scattered electromagnetic wave as well as on the frequency of the spin wave.