In a recent work, Blanc, Le Bris, and Lions defined a notion of increment-stationarity for random point sets, which allowed them to prove the existence of a thermodynamic limit for two-body potential energies on such point sets (under the additional assumption of ergodicity), and to introduce a variant of stochastic homogenization for increment-stationary coefficients. Whereas stationary random point sets are increment-stationary, it is not clear a priori under which conditions increment-stationary random point sets are stationary. In the present contribution, we give a characterization of the equivalence of both notions of stationarity based on elementary PDE theory in the probability space. This allows us to give conditions on the decay of a covariance function associated with the random point set, which ensure that increment-stationary random point sets are stationary random point sets up to a random translation with bounded second moment in dimensions $d>2$. In dimensions $d=1$ and $d=2$, we show that such sufficient conditions cannot exist.