In remote sensing, Single Image Super-Resolution can be learned from large cross-sensor datasets with matched High Resolution and Low Resolution satellite images, thus avoiding the domain gap issue that occurs when generating the Low Resolution image by degrading the High Resolution one. Yet cross-sensor datasets come with their own challenges, caused by the radiometric and geometric discrepancies that arise from using different sensors and viewing conditions. While those discrepancies can be prominent, their impact has been vastly overlooked in the literature, which often focuses on pursuing more complex models without questioning how they can be trained and fairly evaluated in a cross-sensor setting. This paper intends to fill this gap and provide insight on how to train and evaluate cross-sensor Single-Image Super-Resolution Deep Learning models. First, it investigates standard Image Quality metrics robustness to discrepancies and highlights which ones can actually be trusted in this context. Second, it proposes a complementary set of Frequency Domain Analysis based metrics that are tailored to measure spatial frequency restoration performances. Metrics tailored for measuring radiometric and geometric distortion are also proposed. Third, a robust training and evaluation strategy is proposed, with respect to discrepancies. The effectiveness of the proposed strategy is demonstrated by experiments using two widely used cross-sensor datasets: Sen2Venµs and Worldstrat. Those experiments also showcase how the proposed set of metrics can be used to achieve a fair comparison of different models in a cross-sensor setting.