Relative distances between a high-redshift sample of Type Ia supernovae (SNe~Ia), anchored to a low-redshift sample, have been instrumental in drawing insights on the nature of the dark energy driving the accelerated expansion of the universe. A combination (hereafter called SBC) of the SNe~Ia with baryon acoustic oscillations (BAO) from the Dark Energy Spectroscopic Instrument (DESI) and the cosmic microwave background (CMB) recently indicated deviations from the standard interpretation of dark energy as a cosmological constant. In this paper, we analyse various systematic uncertainties in the distance measurement of SNe~Ia and their impact on the inferred dark energy properties in the canonical Chevallier-Polarski-Linder (CPL) model. We model systematic effects like photometric calibration, progenitor and dust evolution, and uncertainty in the galactic extinction law. We find that all the dominant systematic errors shift the dark energy inference towards the DESI 2024 results from an underlying $\Lambda$CDM cosmology. A small change in the calibration, and change in the Milky Way dust, can give rise to systematic-driven shifts on $w_0$-$w_a$ constraints, comparable to the deviation reported from the DESI 2024 results. We forecast that the systematic uncertainties can shift the inference of $w_0-w_a$ by a few times the error ellipse for future low- and high-$z$ SN~Ia compilations and hence, it is critical to circumvent them to robustly test for deviations from $\Lambda$. A slider and visualisation tool for quantifying the impact of systematic effects on the fitted cosmological parameters is publicly available at: https://github.com/sdhawan21/DEslider.git