Geometrical defects are known to have a detrimental influence on the high cycle fatigue resistance of metallic alloys, smaller defects being less harmful. In this experimental work, the influence of the defect size on the high cycle fatigue behavior of polycrystalline aluminium with different grain sizes is investigated, to better understand the role of internal length scales. Firstly, different thermomechanical treatments are applied to obtain aluminium samples with either small (100 μm) or large (1000 μm) grains. The samples are used for preparing fatigue specimens, with either small (100 μm) or large (1000 μm) hemispherical defects. Fully reversed stress-controlled fatigue tests are then carried out. According to fatigue test results, surface crack initiation is delayed when the grain size is reduced, while an approximation of the fatigue limit shows that it is not much influenced by the average grain size. The influence of grain size seems to be explained by the role of cyclic plasticity in the crack initiation process. Finally, Electron BackScattered Diffraction (EBSD) maps are collected for specimens with large grains and small defects. Based on this experimental dataset, fatigue crack initiation from a defect is found to be strongly impacted by the crystallographic orientation of the surrounding grain, crack initiation preferably occurring in crystals being favorably oriented for plastic slip.