Timing of the Crab and Vela pulsars has recently revealed very peculiar evolutions of their spin frequency during the early stage of a glitch. We show that these differences can be interpreted from the interactions between neutron superfluid vortices and proton fluxoids in the core of these neutron stars. In particular, pinning of individual vortices to fluxoids is found to have a dramatic impact on the mutual friction between the neutron superfluid and the rest of the star. The number of fluxoids attached to vortices turns out to be a key parameter governing the global dynamics of the star. These results may have implications for the interpretation of other astrophysical phenomena such as pulsar-free precession or the r-mode instability.