Hyperspectral remote sensing is now an established tool to determine shallow water properties over large areas, usually by inverting a semi-analytical model of water reflectance. However, various sources of error may make the observed subsurface remote-sensing reflectance deviate from the model, resulting in an increased retrieval error when inverting the model based on classical least-squares fitting. In this paper, we propose a probabilistic forward model of shallow water reflectance variability that describes two of the main sources of error, namely, (1) the environmental noise that includes every source of above-water variability (e.g., sensor noise and rough water surface), and (2) the potentially complex inherent spectral variability of each benthic class through their associated spectral covariance matrix. Based on this probabilistic model, we derive two inversion approaches, namely, MILE (MaxImum Likelihood estimation including Environmental noise) and MILEBI (MaxImum Likelihood estimation including Environmental noise and Bottom Intra-class variability) that utilize the information contained in the proposed covariance matrices to further constrain the inversion while allowing the observation to differ from the model in the less reliable wavebands. In this paper, MILE and MILEBI are compared with the widely used least-squares (LS) criterion in terms of depth, water clarity and benthic cover retrievals. For these three approaches, we also assess the influence of constraining bottom mixture coefficients to sum to one on estimation results. The results show that the proposed probabilistic model is a valuable tool to investigate the influence of bottom intra-class variability on subsurface reflectance, e.g., as a function of optical depth or environmental noise. As expected, this influence is critical in very optically shallow waters, and decreases with increasing optical depth. The inversion results obtained from synthetic and airborne data of Quiberon Peninsula, France, show that MILE and MILEBI generally provide better performances than LS. For example, in the case of airborne data with depth ranging from 0.44 to 12.00 m, the bathymetry estimation error decreases by about 32% when using MILE and MILEBI instead of LS. Estimated maps of bottom cover are also more consistent when derived using sum-to-one constrained versions of MILE and MILEBI. MILE is shown to be a simple but powerful method to map simple benthic habitats with negligible influence of intra-class variability. Alternatively, MILEBI is to be preferred if this variability cannot be neglected, since taking bottom covariance matrices into account concurrently with mean reflectance spectra may help the bottom discrimination, e.g., in the presence of overlapping classes. This study thus shows that taking potential sources of error into account through appropriate parameterizations of spectral covariance may be critical to improve the remote sensing of shallow waters, hence making MILE and MILEBI interesting alternatives to LS.