Different mineral fillers from various sources in Portugal were compared as such, in mastics and in asphalt mixtures. The materials included recovered mineral fillers of varying petrographical origin (limestone, granite or rhyolite) obtained from the dedusting system of local asphalt plants, and commercial fillers, active or not, such as limestone filler, hydrated lime, natural hydraulic lime and Portland cement. All fillers were compared using standard filler tests, including dry porosity (Rigden air voids—EN 1097-4), bitumen number (EN 13179-2) and delta Ring and Ball (EN 13179-1). The impact of the filler on the chemical ageing of the binder was also studied using the Pressure Aging Vessel (PAV) test. Finally, dense and porous asphalt mixtures were manufactured with some selected fillers. Their resistance to moisture damage was evaluated using respectively the Indirect Tensile Strength Retained (ITSR - EN 12697-12) and the Cantabro test (EN 12697-11 and NLT-362). From this thorough study, the effect of the different fillers can be compared on all aspects of their potential impact on mixture properties: resistance to moisture damage, aging and mechanical properties. The benefit of using an active filler is clearly demonstrated, especially in the case of hydrated lime.