The purpose of this article is to lay the mathematical foundations of a well known numerical approach in computational statistical physics and molecular dynamics, namely the parallel replica dynamics introduced by A.F. Voter. The aim of the approach is to efficiently generate a coarse-grained evolution (in terms of state-to-state dynamics) of a given stochastic process. The approach formally consists in concurrently considering several realizations of the stochastic process, and tracking among the realizations that which, the soonest, undergoes an important transition. Using specific properties of the dynamics generated, a computational speed-up is obtained. In the best cases, this speed-up approaches the number of realizations considered. By drawing connections with the theory of Markov processes and, in particular, exploiting the notion of quasi-stationary distribution, we provide a mathematical setting appropriate for assessing theoretically the performance of the approach, and possibly improving it.