In an Anderson localized system, a quantum particle with a nonzero initial velocity returns, on average, to its origin. This recently discovered behavior is known as the quantum boomerang effect. Time-reversal invariance was initially thought to be a necessary condition for the existence of this phenomenon. We theoretically analyze the impact of the symmetry breaking on the phenomenon using a one-dimensional system with a spin-orbit coupling and show that the time-reversal invariance is not necessary for the boomerang effect to occur. We explain this behavior giving sufficient symmetry conditions for the boomerang effect to occur when time-reversal symmetry is broken.