We present a comprehensive analysis of all currently available high-accuracy frequency measurements of rotational and rovibrational transitions in the hydrogen molecular ion HD$^+$. Our analysis utilizes the theoretically calculated hyperfine structure to extract the values of three spin-averaged transition frequencies through a global linear least-squares adjustment that takes into account theory-induced correlations between the different transitions. We subsequently use the three spin-averaged transition frequencies as input data in a second adjustment which employs precise theoretical expressions for the transition frequencies, written as a function of the proton, deuteron and electron masses, the Rydberg constant, and the proton and deuteron charge radii. Our analysis shows that the HD$^+$ data may significantly improve the value of the proton-electron mass ratio, in particular if combined with recent high-precision measurements of particle masses and mass ratios obtained from Penning traps.