Helium is one of the first elements clearly identified in the lunar exosphere (Hoffman et al. 1973). Apollo 17 measured the He density at the surface during four lunations. It confirmed the expected day to night asymmetry of the He exosphere with a maximum density near the dawn terminator on the nightside. Few years later, the first detection of Mercury's He exosphere was successfully obtained by Mariner 10 (Broadfoot et al. 1976). These observations highlighted similar global distribution of the He exosphere at Mercury and at the Moon, but also significant differences that have never been convincingly explained. In this paper, we model the He exosphere at the Moon and Mercury with the same approach. The energy accommodation of the exospheric He particles interacting with the surface can be roughly constrained using Apollo 17 and Mariner 10 measurements. Neither a low energy accommodation, as suggested by Shemansky and Broadfoot (1977), nor a full energy accommodation, as suggested by Hodges (1980), can fit all the observations. These observations and their modeling suggest a diurnal variation of the energy distribution of the He ejected from the surface that cannot be explained satisfactorily by any of the present theories on the gas-surface interaction in surface-bounded exospheres.