We present a series of numerical simulations using a shock physics smoothed particle hydrodynamics code, investigating energetic impacts on small celestial bodies characterized by diverse internal structures, ranging from weak and homogeneous compositions to rubble-pile structures with varying boulder volume packing. Our findings reveal that the internal structure of these rubble-pile bodies significantly influences the impact outcomes. Specifically, we observe that the same impact energy can either catastrophically disrupt a target with a low boulder packing (≲30 vol%), or result in the ejection of only a small fraction of material from a target with the same mass but high boulder packing (≳40 vol%). This finding highlights the pivotal role played by the rubble-pile structure, effectively acting as a bulk shear strength, which governs the size and behavior of the resulting impact. Consequently, understanding and characterizing the internal structure of asteroids will be of paramount importance for any future efforts to deflect or disrupt an asteroid on a collision course with Earth.