Can image compression rely on CLIP? - Archive ouverte HAL
Article Dans Une Revue IEEE Access Année : 2024

Can image compression rely on CLIP?

Résumé

Coding algorithms are usually designed to faithfully reconstruct images, which limits the expected gains in compression. A new approach based on generative models allows for new compression algorithms that can reach drastically lower compression rates. Instead of pixel fidelity, these algorithms aim at faithfully generating images that have the same high-level interpretation as their inputs. In that context, the challenge becomes to set a good representation for the semantics of an image. While text or segmentation maps have been investigated and have shown their limitations, in this paper, we ask the following question: do powerful foundation models such as CLIP provide a semantic description suited for compression? By suited for compression, we mean that this description is robust to traditional compression tools and, in particular, quantization. We show that CLIP fulfills semantic robustness properties. This makes it an interesting support for generative compression. To make that intuition concrete, we propose a proof-of-concept for a generative codec based on CLIP. Results demonstrate that our CLIP-based coder beats state-of-the-art compression pipelines at extremely low bitrates ($0.0012$ BPP), both in terms of image quality ($65.3$ for MUSIQ) and semantic preservation ($0.86$ for the Clip score).
Fichier principal
Vignette du fichier
IEEE_Access_Bachard.pdf (30.72 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04600515 , version 1 (04-06-2024)

Licence

Identifiants

Citer

Tom Bachard, Thomas Maugey. Can image compression rely on CLIP?. IEEE Access, 2024, pp.1-17. ⟨10.1109/ACCESS.2024.3408651⟩. ⟨hal-04600515⟩
46 Consultations
6 Téléchargements

Altmetric

Partager

More