Nonlinear interactions in coherent gases are not only at the origin of of bright and dark solitons and superfluids. At the same time, they give rise to phenomena such as multistability, which hold great promise for the development of advanced photonic and spintronic devices. In particular, spinor multistability in strongly-coupled semiconductor microcavities shows that the spin of hundreds of exciton-polaritons can be coherently controlled, opening the route to spin-optronic devices such as ultrafast spin memories, gates or even neuronal communication schemes. Here, we demonstrate that switching between the stable spin states of a driven polariton gas can be controlled by ultrafast optical pulses. While such a long-lived spin memory necessarily relies on strong and anisotropic spinor interactions within the coherent polariton gas, we also highlight the crucial role of nonlinear losses and formation of a non-radiative particle reservoir for ultrafast spin-switching.