The $p$ process nucleosynthesis is responsible for the synthesis of 35 neutron-deficient nuclei from $^{35}$Se to $^{196}$Hg. An important input that can affect the modeling of this process is the nuclear level density at the relevant excitation energies of the nuclei involved in the reaction network. The OSLO method has been extensively used for the measurement of level densities in excitation energies of several MeV. In this work, Bayesian optimization has been used in order to estimate the 95% high density intervals for the parameters of two level density models optimized on the OSLO data. These uncertainties are then propagated on the cross sections of $(p,\gamma)$ reactions leading to the compound nuclei $^{105,106}$Pd and $^{105,106}$Cd inside the astrophysically relevant energy range. Imposing constraints in this region of the isotopic chart is important for network calculations involving the nearby $p$ nuclei $^{102}$Pd and $^{106}$Cd. We discuss the reduction of the range of cross sections due to the uncertainties arising from the level density data compared to the range of the six default level density models available in TALYS and we highlight the need for level density data inside the astrophysically relevant energy ranges.