Polymer nanocomposites (PNCs) are known to display exceptional non-linear mechanical properties, even for deformation amplitudes as low as a few percent. One of the most studied phenomenon is the so-called Payne effect, a drop in storage modulus with increasing amplitude. Several mechanisms have been put forward to rationalize such effect, including filler network breaking, polymer chain desorption and yielding of the polymer confined between the nanofillers. In this contribution, we demonstrate that for PNCs involving fractal-like aggregates, the Payne effect may originate from the alignment of the aggregates under the imposed flow direction. We reach this conclusion by using a coarse-grained model, which combines an explicit representation of fillers with an implicit description of the polymer matrix. We systematically characterize the effects of aggregate size and polydispersity in the amplitude of the Payne effect. Moreover, we probe the mechanical response of the model PNCs after a first cycle of deformation. We observe slow recovery kinetics of the original storage modulus of the PNCs and relate this memory effect to the alignment of the aggregates. Our findings should contribute to clarify the relation between the macroscopic mechanical response of the PNCs and the mesoscopic state of the filler.