Abstract Ground-based thermal infrared observations face substantial challenges in correcting the predominant background emitted as thermal radiation from the atmosphere and the telescope itself. With the upcoming 40 m class ELTs, unprecedented sensitivities from ground will be reached, underlining the need of even more sophisticated background correction strategies. This study aims to investigate the impact of thermal backgrounds on ground-based observations and identify possible limiting factors in dedicated correction strategies. We evaluate temporal and spatial characteristics of the thermal background in direct imaging data obtained with different telescopes and observation modes. In particular, three distinct data sets, acquired using VLT/NACO and KECK/NIRC2, are analyzed. Our analysis reveals that the observations are not fully photon shot noise limited, but exhibit additional sensitivity losses caused by imperfect background compensation in the different data sets. We identify correlations between background fluctuations and the activity of the adaptive optics system. We hypothesize that the pupil modulation of the adaptive optics mirrors introduces high frequency spatial and temporal fluctuations to the background, which could ultimately constrain the detection limit if they are not compensated adequately.