Micro-Doppler analysis commonly makes use of the log-scaled, real-valued spectrogram, and recent work involving deep learning architectures for classification are no exception. Some works in neighboring fields of research directly exploit the raw temporal signal, but do not handle complex numbers, which are inherent to radar IQ signals. In this paper, we propose a complex-valued, fully temporal neural network which simultaneously exploits the raw signal and the spectrogram by introducing a Fourier-like layer suitable to deep architectures. We show improved results under certain conditions on synthetic radar data compared to a real-valued counterpart.