Keystroke dynamics allows to authenticate individuals through their way of typing their password or a free text on a keyboard. In general, in biometrics, a novel algorithm is validated through a comparison to the state of the art one's using some datasets in an offline way. Several benchmark datasets for keystroke dynamics have been proposed in the literature. They differ in many ways and their intrinsic properties influence the performance of the algorithms under evaluation. In this work, we (a) provide a literature review on existing benchmark datasets of keystroke dynamics; (b) present several criteria and tests in order to characterize them; (c) and apply these criteria on these available public benchmark datasets. The review analysis shows a great disparity in the acquisition protocol, the population involved, the complexity of the passwords, or the expected performance (there is a relative difference of 76% between the EER on the worst and best performing datasets with the same authentication method).