In a Josephson junction, the transfer of Cooper pairs from one superconductor to the other one can be associated with the formation of Andreev bound states. In a Josephson junction made with a semiconducting nanowire, the spin degeneracy of these Andreev states can be broken thanks to the presence of spin-orbit coupling and a finite phase difference between the two superconducting electrodes. The lifting of the spin degeneracy opened the way to the realization of Andreev spin qubits that do not require the application of a large magnetic field. So far the operation of these qubits relied on a Raman process involving two microwave tones and a third Andreev state [16M. Hays , Science 373, 430 (2021)]. Still, time-reversal preserving impurities in the nanowire allow for spin-flip scattering processes. Here, using the formalism of scattering matrices, we show that these processes generically couple Andreev states with opposite spins. In particular, the nonvanishing current matrix element between them allows for the direct manipulation of phase-driven Andreev spin qubits, thereby circumventing the use of the above-mentioned Raman process.