This work discusses how to choose performance measures to compare numerical simulations of a flood event with one satellite image, e.g., in a model calibration or validation procedure. A series of criterion are proposed to evaluate the sensitivity of performance measures with respect to the flood extent, satellite characteristics (position, orientation), and measurements/processing errors (satellite raw values or extraction of the flood maps). Their relevance is discussed numerically in the case of one flooding event (on the Garonne River in France in February 2021), using a distribution of water depths simulated from a shallow-water model parameterized by an uncertain friction field. After identifying the performance measures respecting the most criteria, a correlation analysis is carried out to identify how various performance measures are similar. Then, a methodology is proposed to rank performance measures and select the most robust to observation errors. The methodology is shown useful at identifying four performance measures out of 28 in the study case. Note that the various top-ranked performance measures do not lead to the same calibration result as regards the friction field of the shallow-water model. The methodology can be applied to the comparison of any flood model with any flood event.