Most of crystalline materials exhibit a hysteresis on their deformation curve when mechanically loaded in alternating directions. This Bauschinger effect is the signature of mechanisms existing at the atomic scale and controlling the materials damage and ultimately their failure. Here, three-dimensional simulations of dislocation dynamics and statistical analyses of the microstructure evolution reveal two original elementary mechanisms. An asymmetry in the dislocation network junctions arising from the stress driven curvatures and the partial reversibility of plastic avalanches give an explanation to the traction-compression asymmetry observed in FCC single-crystals. These mechanisms are then connected in a physically justified way to larger-scale representations using a dislocation density based theory. Parameter-free predictions of the Bauschinger effect and strain hardening during cyclic deformation in different materials and over a range of loading directions and different plastic strain amplitudes are found to be in excellent agreement with experiments. This work brings invaluable mechanistic insights for the interpretation of experiments and for the design of structural components to consolidate their service life under cyclic load.