Article Dans Une Revue Transactions on Machine Learning Research Journal Année : 2024

VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing

Résumé

Recently, diffusion-based generative models have achieved remarkable success for image generation and edition. However, their use for video editing still faces important limitations. This paper introduces VidEdit, a novel method for zero-shot text-based video editing ensuring strong temporal and spatial consistency. Firstly, we propose to combine atlas-based and pre-trained text-to-image diffusion models to provide a training-free and efficient editing method, which by design fulfills temporal smoothness. Secondly, we leverage off-the-shelf panoptic segmenters along with edge detectors and adapt their use for conditioned diffusion-based atlas editing. This ensures a fine spatial control on targeted regions while strictly preserving the structure of the original video. Quantitative and qualitative experiments show that VidEdit outperforms state-of-the-art methods on DAVIS dataset, regarding semantic faithfulness, image preservation, and temporal consistency metrics. With this framework, processing a single video only takes approximately one minute, and it can generate multiple compatible edits based on a unique text prompt.

Fichier principal
Vignette du fichier
2306.08707v4.pdf (44.13 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04310067 , version 1 (28-01-2026)

Licence

Identifiants

Citer

Paul Couairon, Clément Rambour, Jean-Emmanuel Haugeard, Nicolas Thome. VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing. Transactions on Machine Learning Research Journal, 2024. ⟨hal-04310067⟩
181 Consultations
10 Téléchargements

Altmetric

Partager

  • More