We use very long baseline interferometry to measure the proper motions of three black hole X-ray binaries (BHXBs). Using these results together with data from the literature and Gaia DR2 to collate the best available constraints on proper motion, parallax, distance, and systemic radial velocity of 16 BHXBs, we determined their three-dimensional Galactocentric orbits. We extended this analysis to estimate the probability distribution for the potential kick velocity (PKV) a BHXB system could have received on formation. Constraining the kicks imparted to BHXBs provides insight into the birth mechanism of black holes (BHs). Kicks also have a significant effect on BH–BH merger rates, merger sites, and binary evolution, and can be responsible for spin–orbit misalignment in BH binary systems. 75 per cent of our systems have potential kicks |$\gt 70\, \rm {km\,s^{-1}}$|⁠. This suggests that strong kicks and hence spin–orbit misalignment might be common among BHXBs, in agreement with the observed quasi-periodic X-ray variability in their power density spectra. We used a Bayesian hierarchical methodology to analyse the PKV distribution of the BHXB population, and suggest that a unimodal Gaussian model with a mean of 107 |$\pm \,\,16\, \rm {km\,s^{-1}}$| is a statistically favourable fit. Such relatively high PKVs would also reduce the number of BHs likely to be retained in globular clusters. We found no significant correlation between the BH mass and PKV, suggesting a lack of correlation between BH mass and the BH birth mechanism. Our python code allows the estimation of the PKV for any system with sufficient observational constraints.