Past primary productivity is tracked in lake sediments to reconstruct environmental changes. However, the resolution of the routinely used destructive techniques is not suitable for the analysis of a large number of samples due to cost. Non-destructive analysis of chlorophyll-a performed by hyperspectral imagery enables the quick determination of indices of past primary productivity. Eighteen indices used in paleo-environmental reconstruction were inventoried to define the best index capable of tracking chlorophyll-a by this technique. All the indices were applied to hyperspectral data measured on the sediment of Lake Bresson, in which detrital organic matter input is likely to skew chlorophyll-a identification, and the results were compared with concentrations measured by a routinely used destructive analysis. The 18 indices all produced a different result and only three indices were positively correlated with chlorophyll-a concentrations (n = 28, p < 0.0001). The detrital organic matter impacts chlorophyll-a characterization and shows the bias produced by the sediment matrix variations. Moreover, artificial modification of the sediment matrix revealed an impact of the mineral phase. To tackle this issue, the indices are normalized by two proxies of the sediment components. This new approach reduces the impact of detrital organic matter, hence the sediment matrix variations also reduce the normalization of the chlorophyll-a indices by a specific proxy of the mineral phase. These results identify the impact of local geochemical features that limit past primary productivity assessment and show the necessity of characterizing the sediment composition prior to tracking the chlorophyll-a by hyperspectral imaging.