The structure of the asteroid belt holds a record of the Solar System's dynamical history. The current belt only contains 10-3 Earth masses yet the asteroids' orbits are dynamically excited, with a large spread in eccentricity and inclination. The belt is also chemically segregated: the inner belt is dominated by dry S-types and the outer belt by hydrated C-types. Here we propose a new model in which the asteroid belt was always low-mass and was partially populated and sculpted by the giant planets on chaotic, resonant orbits. We first show that the compositional dichotomy of the asteroid belt is a simple consequence of Jupiter's growth in the gaseous protoplanetary disk. As Jupiter's core rapidly grew by accreting gas, orbits of nearby planetesimals were perturbed onto Jupiter-crossing trajectories. A significant fraction (~10%) of objects in the neighborhood exterior of Jupiter's orbit were implanted by gas drag into the outer parts of the asteroid belt as C-types. While the gas giants were likely in mean motion resonance at the end of the gaseous disk phase, we show that small perturbations may have driven them into a chaotic but stable state. After the dissipation of the gaseous disk, stochastic variations in the gas giants orbits caused resonances to chaotically jump across the main belt and excite the asteroids' orbits. Our results suggest that the early Solar System was chaotic and introduce a simple framework to understand the origins of the asteroid belt.