The solar system's dynamical state can be explained by an orbital instability among the giant planets. A recent model has proposed that the giant planet instability happened during terrestrial planet formation. This scenario has been shown to match the inner solar system by stunting Mars' growth and preventing planet formation in the asteroid belt. Here we present a large sample of new simulations of the "Early Instability" scenario. We use an N-body integration scheme that accounts for collisional fragmentation, and also perform a large set of control simulations that do not include an early giant planet instability. Since the total particle number decreases slower when collisional fragmentation is accounted for, the growing planets' orbits are damped more strongly via dynamical friction and encounters with small bodies that dissipate angular momentum (eg: hit-and-run impacts). Compared with simulations without collisional fragmentation, our fully evolved systems provide better matches to the solar system's terrestrial planets in terms of their compact mass distribution and dynamically cold orbits. Collisional processes also tend to lengthen the dynamical accretion timescales of Earth analogs, and shorten those of Mars analogs. This yields systems with relative growth timescales more consistent with those inferred from isotopic dating. Accounting for fragmentation is thus supremely important for any successful evolutionary model of the inner solar system.