We study rating systems, such as the famous ELO system, applied to a large number of players. We assume that each player is characterized by an intrinsic inner strength and follow the evolution of their rating evaluations by deriving a new continuous model, a kinetic-like equation. We then investigate the validity of the rating systems by looking at their large time behavior as one would ideally expect the rating of each player to converge to their actual strength. The simplistic case when all players interact indeed yields an exponential convergence of the ratings. However, the behavior in the more realistic cases with only local interactions is more complex with several possible equilibria depending on the exact initial distribution of initial ratings and possibly very slow convergence.