Communication Dans Un Congrès Année : 2025

PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models

Résumé

Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments.

Fichier principal
Vignette du fichier
2504.08966v1.pdf (1.57 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05052399 , version 1 (30-04-2025)

Licence

Identifiants

Citer

Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, Aymen Shabou. PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025), Jun 2025, Nashville, TN, United States. ⟨hal-05052399⟩
200 Consultations
113 Téléchargements

Altmetric

Partager

  • More