Specialists species thriving under specific environmental conditions in narrow geographic ranges are widely recognised as heavily threatened by climate deregulation. Many might rely on both their potential to adapt and to disperse toward a refugium to avoid extinction. It is thus crucial to understand the influence of environmental conditions on the unfolding process of adaptation. Here, I study the eco-evolutionary dynamics of a sexually reproducing specialist species in a two-patch quantitative genetic model with moving optima. Thanks to a separation of ecological and evolutionary time scales and the phase-line study of the selection gradient, I derive the critical environmental speed for persistence, which reflects how the existence of a refugium impacts extinction patterns. Moreover, the analysis provides key insights about the dynamics that arise on the path towards this refugium. I show that after an initial increase of population size, there exists a critical environmental speed above which the species crosses a tipping point, resulting into an abrupt habitat switch. Besides, when selection for local adaptation is strong, this habitat switch passes through an evolutionary "death valley", leading to a phenomenon related to evolutionary rescue, that can promote extinction for lower environmental speeds than the critical one.