Background: Caffeine is a well-known psychostimulant reputed to alleviate the deleterious effects of sleep deprivation. Nevertheless, caffeine can alter sleep duration and quality, particularly during recovery sleep. We evaluated the effects of acute caffeine intake on the duration and quality of recovery sleep following total sleep deprivation (TSD), taking into account daily caffeine consumption. Methods: Forty-one participants performed a double-blind, crossover TSD protocol (38 h of continuous wakefulness) with acute caffeine or placebo. Caffeine (2.5 mg/kg) or placebo was administered twice during continuous wakefulness (last treatment 6.5 h before bedtime for the recovery night). Polysomnographic measurements were recorded using a connected headband. Results: TSD was associated with a rebound in total sleep time (TST) on the recovery night (+110.2 ± 23.2 min, p < 0.001). Caffeine intake decreased this recovery TST (−30.2 ± 8.2 min p = 0.02) and the N3 sleep stage duration (−35.6 ± 23.2 min, p < 0.01). Caffeine intake altered recovery sleep continuity (increased number of long awakenings), stability (higher stage transition frequency), and organization (less time spent in complete sleep cycle) and decreased the delta power spectral density during NREM sleep. On the recovery night, habitual daily caffeine consumption was negatively correlated with TST in caffeine and placebo conditions and positively correlated with wake after sleep onset (WASO) duration and with the frequency of long (>2 min) awakenings in the caffeine condition only. Conclusions: Acute caffeine intake during TSD affects nighttime recovery sleep, with an interaction with daily consumption. These results may influence advice on caffeine intake for night-shift workers. (NCT03859882).