Frequency and dc magnetic field dependences of dynamic susceptibility in diluted paramagnets LiYF$_4$:Ho$^{3+}$ have been measured at liquid helium temperatures in the ac and dc magnetic fields parallel to the symmetry axis of a tetragonal crystal lattice. Experimental data are analyzed in the framework of microscopic theory of relaxation rates in the manifold of 24 electron-nuclear sublevels of the lowest non-Kramers doublet and the first excited singlet in the Ho$^{3+}$ ground multiplet $^5I_8$ split by the crystal field of S$_4$ symmetry. The one-phonon transition probabilities were computed using electron-phonon coupling constants calculated in the framework of exchange charge model and were checked by optical piezospectroscopic measurements. The specific features observed in field dependences of the in- and out-of-phase susceptibilities (humps and dips, respectively) at the crossings (anti-crossings) of the electron-nuclear sublevels are well reproduced by simulations when the phonon bottleneck effect and the cross-spin relaxation are taken into account.