Several thermally activated mechanisms have been proposed to explain the dynamic weakening of faults at fast, seismic slip rates v > 0.1 m s-1. Nevertheless, direct constraints from microstructural observations on the operation of such weakening mechanisms are still lacking for most rock forming minerals. This includes olivine, a very refractory mineral relevant for seismic activity occurring in peridotite massifs. Here, we report mechanical data and microstructures of olivine aggregates deformed in a rotary shear apparatus at slip rates from 10-2 m s-1 to 1 m s-1. A number of 34 shear deformation experiments were performed under axial stresses of 20 MPa and at room temperature. Samples are composed of either synthetic iron-free nanocrystalline forsterite powder (initial grain size = 0.07 µm) or natural iron-bearing olivine powder from San Carlos or San Bernardino (Fo91, Arizona, USA; initial grain size = 70 ± 2 µm). Shear deformation lasts from a few seconds up to 5 minutes and ends after 0.03 to 3 m of slip. Independently of the initial grain size or composition, friction coefficients decrease from peak values of 0.8 to values below 0.4, after only 0.1 m of slip. The recovered samples were characterized by scanning electron microscopy, electron backscatter diffraction and transmission electron microscopy. For all samples, before the onset of dynamic weakening, the deformation localizes in a principal shear zone, well-marked in the natural olivine aggregates by a rapid decrease of grain size down to < 1.7 µm. Variations in texture, fractured grains and the formation of a fine-grained matrix support cataclastic flow at an early stage of deformation, responsible for the foremost grain size reduction. The samples display well-defined zones with partially sintered microstructure, and shape preferred orientation. The initial texture of the cold-pressed starting material is rapidly reduced to a near random distribution after only 0.03 m of slip. All crystal preferred orientations (CPO) remain weak, with a J-index ≲ 2. Based on the microstructures, the dominant deformation mechanism may be enhanced effective diffusion (by surface and grain boundary diffusion) causing mechanical weakening and a viscous behavior, which is then temperature-dependent under sub-solidus conditions. We find no microstructural evidence for frictional melting, flash heating or thermal decomposition. Our results advocate that dynamic weakening of faults is independent of initial grain size, and can be achieved after displacements as low as 0.1 m without production of frictionally induced-melt.