Several studies have used grape harvest date (GHD) as a proxy for temperature variations of the last centuries in Europe. However, the use of grape harvest dates to reconstruct climate is not straightforward, with four possible causes of major flaws. In this study we identify and evaluate the accuracy of GHD as a proxy to past temperature anomalies, uncertainties in the model used to relate temperature to GHD, identity of the grape varieties cultivated in the past, type of wine produced in the past and cultural practices used in the past. Our analyses are based on several phenological and crop models, and on the most complete data set on grape vine phenology and harvest quality. We show that the two methodologies currently used — linear regression models and process-based phenological models — can be accurate, but process-based phenological models ascertain robustness to be applied confidently in different vineyards and different periods. However, we show that several factors can induce a bias in temperature reconstructions using process-based models. We demonstrate the importance of historical information on the studied areas such as the varieties cultivated, the style of wine produced, the quality sought, the agricultural practices, in order to build the most robust model.