Passive ultrasound imaging is of great interest for cavitation monitoring. Spatio-temporal monitoring of cavitation bubbles in therapeutic applications is possible using an ultrasound imaging probe to passively receive the acoustic signals from the bubbles. Fourier-domain beamformers have been proposed to process the signals received into maps of the spatial localization of cavitation activity, with reduced computing times with respect to the time-domain approach, and to take advantage of frequency selectivity for cavitation regime characterization. The approaches proposed have been mainly nonadaptive, and these have suffered from low resolution and contrast, due to the many reconstruction artifacts. Inspired by the array-processing literature, and in the context of passive ultrasound imaging of cavitation, we propose here a robust estimation of the second-order statistics of data through spatial covariance matrices in the Fourier domain, or cross-spectral density matrices. The benefits of such formalism are illustrated using advanced reconstruction algorithms, such as the robust Capon beamformer, the Pisarenko class beamformer, and the multiple signal classification approach. Through both simulations and experiments in a water tank, we demonstrate that enhanced localization of cavitation activity (i.e., improved resolution and contrast with respect to nonadaptive approaches) is compatible with the rapid and frequency-selective approaches of the Fourier domain. Robust estimation of the cross-spectral density matrix and the derived adaptive beamformers pave the way to the development of powerful passive ultrasound imaging tools.