Quantitative ultrasound can be used to characterize the evolution of the bone-implant interface (BII), which is a complex system due to the implant surface roughness and to partial contact between bone and the implant. The aim of this study is to derive the main determinants of the ultrasonic response of the BII during osseointegration phenomena. The influence of (i) the surface roughness parameters and (ii) the thickness W of a soft tissue layer on the reflection coefficient r of the BII was investigated using a two-dimensional finite element model. When W increases from 0 to 150 μm, r increases from values in the range [0.45; 0.55] to values in the range [0.75; 0.88] according to the roughness parameters. An optimization method was developed to determine the sinusoidal roughness profile leading to the most similar ultrasonic response for all values of W compared to the original profile. The results show that the difference between the ultrasonic responses of the optimal sinusoidal profile and of the original profile was lower to typical experimental errors. This approach provides a better understanding of the ultrasonic response of the BII, which may be used in future numerical simulation realized at the scale of an implant.