Soft particulate gels include materials we can eat, squeeze, or 3D print. From foods to bio-inks to cement hydrates, these gels are composed of a small amount of particulate matter (proteins, polymers, colloidal particles, or agglomerates of various origins) embedded in a continuous fluid phase. The solid components assemble to form a porous matrix, providing rigidity and control of the mechanical response, despite being the minority constituent. The rheological response and gel elasticity are direct functions of the particle volume fraction φ: however, the diverse range of different functional dependencies reported experimentally has, to date, challenged efforts to identify general scaling laws. Here we reveal a hidden hierarchical organization of fractal elements that controls the viscoelastic spectrum, and which is associated with the spatial heterogeneity of the solid matrix topology. The fractal elements form the foundations of a viscoelastic master curve, which we construct using large-scale 3D microscopic simulations of model gels, and can be described by a recursive rheological ladder model over a range of particle volume fractions and gelation rates. The hierarchy of the fractal elements provides the missing general framework required to predict the gel elasticity and the viscoelastic response of these ubiquitous complex materials.