The high visual quality of modern deepfakes raises significant concerns about the trustworthiness of digital media and makes facial tampering detection more challenging. Although current deep learning-based deepfake detectors achieve excellent results when tested on deepfake images or image sequences generated using known methods, generalization—where a trained model is tasked with detecting deepfakes created with previously unseen manipulation techniques—is still a major challenge. In this paper, we investigate the impact of training spatial and spatio-temporal deep learning network architectures in the image noise residual domain using spatial rich model (SRM) filters on generalization performance. To this end, we conduct a series of tests on the manipulation methods of the FaceForensics++, DeeperForensics-1.0 and Celeb-DF datasets, demonstrating the value of image noise residuals and temporal feature exploitation in tackling the generalization task.