We present an exact solution for the probability density function $P(\tau=t_{\min}-t_{\max}|T)$ of the time-difference between the minimum and the maximum of a one-dimensional Brownian motion of duration $T$. We then generalise our results to a Brownian bridge, i.e. a periodic Brownian motion of period $T$. We demonstrate that these results can be directly applied to study the position-difference between the minimal and the maximal height of a fluctuating $(1+1)$-dimensional Kardar-Parisi-Zhang interface on a substrate of size $L$, in its stationary state. We show that the Brownian motion result is universal and, asymptotically, holds for any discrete-time random walk with a finite jump variance. We also compute this distribution numerically for L\'evy flights and find that it differs from the Brownian motion result.