Aims. The path towards robust near-infrared extensions of stellar population models involves the confrontation between empirical and synthetic stellar spectral libraries across the wavelength ranges of photospheric emission. Indeed, the theory of stellar emission enters all population synthesis models, even when this is only implicit in the association of fundamental stellar parameters with empirical spectral library stars. With its near-ultraviolet to near-infrared coverage, the X-shooter Spectral Library (XSL) allows us to examine to what extent models succeed in reproducing stellar energy distributions (SEDs) and stellar absorption line spectra simultaneously. Methods. As a first example, this study compares the stellar spectra of XSL with those of the Göttingen Spectral Library, which are based on the PHOENIX synthesis code. The comparison was carried out both separately in the three arms of the X-shooter spectrograph known as UVB, VIS and NIR, and jointly across the whole spectrum. We did not discard the continuum in these comparisons; only reddening was allowed to modify the SEDs of the models. Results. When adopting the stellar parameters published with data release DR2 of XSL, we find that the SEDs of the models are consistent with those of the data at temperatures above 5000 K. Below 5000 K, there are significant discrepancies in the SEDs. When leaving the stellar parameters free to adjust, satisfactory representations of the SEDs are obtained down to about 4000 K. However, in particular below 5000 K and in the UVB spectral range, strong local residuals associated with intermediate resolution spectral features are then seen; the necessity of a compromise between reproducing the line spectra and reproducing the SEDs leads to dispersion between the parameters favored by various spectral ranges. We describe the main trends observed and we point out localized offsets between the parameters preferred in this global fit to the SEDs and the parameters in DR2. These depend in a complex way on the position in the Hertzsprung-Russell diagram (HRD). We estimate the effect of the offsets on bolometric corrections as a function of position in the HRD and use this for a brief discussion of their impact on the studies of stellar populations. A review of the literature shows that comparable discrepancies are mentioned in studies using other theoretical and empirical libraries.