The hysteretic softening at small dynamic strains (Payne effect) related to the rolling resistance and viscoelastic losses of tires was studied as a function of particle size, filler volume fraction, and temperature for carbon black (CB) reinforced uncrosslinked styrene-butadiene rubber (SBR) and a paste-like material composed of CB-filled paraffin oil. The low strain limit for dynamic storage modulus was found to be remarkably similar for CB-filled oil compared to CB-filled SBR. Small-angle X-ray scattering (SAXS) measurements on the simple composites and detailed data analysis confirmed that the aggregate structures and nature of filler branching/networking of carbon black were virtually identical within oil compared to the high molecular weight polymer matrix. The combined dynamic rheology and SAXS results provide clear evidence that the deformation-induced breaking (unjamming) of the filler network – characterized by filler-filler contacts that are percolated throughout the material – is the main cause for the Payne effect. However, the polymer matrix does play a secondary role as demonstrated by a reduction in Payne effect magnitude with increasing temperature for the CB-reinforced rubber, which was not observed to a significant extent for the oil-CB system.