Integrating perceptual quality analysis and caption-based features for robust deepfake video detection
Résumé
Inspired by the benefits of both linguistic representations as well as visual quality analysis, we propose a multimodal framework for deepfake detection with reliable accuracy. Our method acquires low-level perceptual features from frames of video, including contrast, brightness, and sharpness, and computes artifact scores for artificial anomaly detection. In parallel, our method produces descriptive frame-level captions, summed up for generating video-level summaries for capturing contextual coherence. For training the model, our method leverages the FaceForensics++ dataset, comprising several techniques of deepFake manipulation, including DeepFake, Face2Face, FaceSwap, as well as NeuralTextures. Metadata comprised of quality measurements, the artifact scores, as well as textual captions, is tokenized, processed by a DeepSeek V2 Lite model, fine-tuned by the Low-Rank Adaptation (LoRA) procedure, as the backbone for classification. We test the proposed model on both intra-dataset as well as cross-dataset settings, demonstrating good accuracy as well as impressive generalization ability. Our method achieves a 96.51% accuracy for classifying, by integrating low-level perceptual as well as high-level semantic reason, effectively marking manipulated content as well as upholding the integrity of the medium in a world of increasingly evolving generative methods. The employed source code as well as dataset preprocessing scripts for our work are made available for public access on: https://github.com/HessenUPHF/perceptual-caption-detection
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| Licence |