We propose a solution method for online vehicle routing, which integrates a machine learning routine to improve tours’ quality. Our optimization model is based on the Bertsimas et al. (2019) re-optimization approach. Two separate routines are developed. The first one uses a neural network to produce realistic pick-up times for the customers to serve. The second one relies on Q-learning in addition to random walks for the construction of the backbone graph corresponding to the instance problem of each time step. The second routine gives improved results compared to the original approach.