On 11 November 2019, an Mw 4.9 earthquake struck the middle Rhône valley (South-East France) producing moderate to severe damage in the town of Le Teil and its surroundings. This unexpected event stressed the vulnerability of the French cultural built heritage to a moderate seismic hazard. Commonly applied to modern civil engineering structures, passive seismic methods are still lacking on historic constructions to understand properly the different factors driving their dynamic behavior. In this article, the results of a two-month seismic monitoring survey carried out shortly after the Le Teil mainshock in a historic masonry tower are presented and discussed. Located only 5 km south of the epicenter, the Gate Tower of Viviers (eleventh century) was instrumented with four highly sensitive seismic nodes. Ambient vibrations, as well as aftershocks and quarry blasts from the nearby Le Teil quarry, were recorded and used in the analysis. Through vibration-based analysis, the article addresses three relevant aspects of the dynamic response of ancient masonry structures. We discuss first the differences in the building’s response induced by the three reported types of vibrations, focusing on the particular signal characteristics of shallow aftershocks and quarry blasts. Then, we apply the Random Decrement Technique (RDT) to track the dynamic behavior variations over two months and to discuss the role of the environmental conditions in the slight fluctuations of the structural modal parameters (natural frequencies, damping coefficients) of unreinforced masonry structures. We also show evidence of the nonlinear elastic behavior under both weak seismic and atmospheric loadings. The correlation between the presence of heterogeneities in the construction materials and the nonlinear threshold supports the relevance of such types of monitoring surveys as a valuable tool for future modeling works and conservation efforts.