The asteroid belt contains less than a thousandth of Earth's mass and is radially segregated, with S-types dominating the inner belt and C-types the outer belt. It is generally assumed that the belt formed with far more mass and was later strongly depleted. Here we show that the present-day asteroid belt is consistent with having formed empty, without any planetesimals between Mars and Jupiter's present-day orbits. This is consistent with models in which drifting dust is concentrated into an isolated annulus of terrestrial planetesimals. Gravitational scattering during terrestrial planet formation causes radial spreading, transporting planetesimals from inside 1-1.5 AU out to the belt. Several times the total current mass in S-types is implanted, with a preference for the inner main belt. C-types are implanted from the outside, as the giant planets' gas accretion destabilizes nearby planetesimals and injects a fraction into the asteroid belt, preferentially in the outer main belt. These implantation mechanisms are simple byproducts of terrestrial- and giant planet formation. The asteroid belt may thus represent a repository for planetary leftovers that accreted across the Solar System but not in the belt itself.