DiffEdit: Diffusion-based Semantic Image Editing with Mask Guidance - Archive ouverte HAL Access content directly
Conference Papers Year : 2023

DiffEdit: Diffusion-based Semantic Image Editing with Mask Guidance

Jakob Verbeek
  • Function : Author
  • PersonId : 1220019
Holger Schwenk
  • Function : Author
  • PersonId : 1220020
Matthieu Cord


Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DIFFEDIT, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. DIFFEDIT achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images.
Fichier principal
Vignette du fichier
DiffEdit_camera_ready_25jan.pdf (19.43 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03957480 , version 1 (26-01-2023)


  • HAL Id : hal-03957480 , version 1


Guillaume Couairon, Jakob Verbeek, Holger Schwenk, Matthieu Cord. DiffEdit: Diffusion-based Semantic Image Editing with Mask Guidance. ICLR 2023 (Eleventh International Conference on Learning Representations ), ICLR, May 2023, Kigali, Rwanda. ⟨hal-03957480⟩
127 View
94 Download


Gmail Facebook X LinkedIn More