Recycled aggregates, especially the fine (0–4 mm) fraction, are often contaminated with sulfates coming from gypsum residues on demolition sites. When these aggregates are used in concrete, the sulfates can induce internal sulfate attack which causes expansion of the concrete. European standard EN 206:2013 sets the water-soluble sulfate limit at 0.2% by weight of the aggregate, but other studies suggest this limit could be safely increased. In addition to the sulfate content, other parameters, such as the porosity and alkalinity of the mix, have been shown to influence the swelling results. In this study, different proposed sulfate limits were evaluated on concrete made with 100% fine recycled aggregates (FRA). It was also researched whether the mixing parameters could change the swelling amount regardless of the sulfate content. The results showed that the incorporation of FRA with sulfate contents up to 0.8 wt% is safe when combined with coarse natural aggregates. If coarse recycled aggregates are used in the mix, the sulfate content of FRA can be up to 0.3%. The swelling caused by these sulfate levels was not high enough to be influenced by the porosity or alkalinity of the mix.