It is generally admitted that in quantum mechanics, the electromagnetic potentials have physical interpretations otherwise absent in classical physics as illustrated by the Aharonov-Bohm effect. In 1984, Berry interpreted this effect as a geometrical phase factor. The same year, Wilczek and Zee generalized the concept of Berry phases to degenerate levels and showed that a non-Abelian gauge field arises in these systems. In sharp contrast with the Abelian case, spatially uniform non-Abelian gauge fields can induce particle noninertial motion. We explore this intriguing phenomenon with a degenerated Fermionic atomic gas subject to a two-dimensional synthetic SU(2) non-Abelian gauge field. We reveal the spin Hall nature of the noninertial dynamic as well as its anisotropy in amplitude and frequency due to the spin texture of the system. We finally draw the similarities and differences of the observed wave packet dynamic and the celebrated Zitterbewegung effect of the relativistic Dirac equation.