Communication Dans Un Congrès Année : 2025

Q-Filters: Leveraging Query-Key Geometry for Efficient Key-Value Cache Compression

Résumé

Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths grow, the KV Cache becomes a significant memory bottleneck, which calls for compression methods that limit its size during generation. In this paper, we discover surprising properties of Query (Q) and Key (K) vectors that allow us to efficiently approximate attention scores without computing the attention maps. We propose Q-Filters, a training-free KV Cache compression method that filters out less crucial Key-Value pairs based on a single context-agnostic projection. Contrarily to many alternatives, Q-Filters is compatible with FlashAttention, as it does not require direct access to attention weights. Experimental results in long-context settings demonstrate that Q-Filters is competitive with attention-based compression methods such as SnapKV in retrieval tasks while consistently outperforming efficient compression schemes such as Streaming-LLM in generation setups. Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task with a x32 compression level while reducing the generation perplexity drop by up to 65% in text generation compared to Streaming-LLM.

Fichier principal
Vignette du fichier
75_Q_Filters_Leveraging_Query_.pdf (4.32 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05441473 , version 1 (17-02-2026)

Licence

Identifiants

  • HAL Id : hal-05441473 , version 1

Citer

Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, et al.. Q-Filters: Leveraging Query-Key Geometry for Efficient Key-Value Cache Compression. ICLR 2025 Workshop on Sparsity in LLMs (SLLM), Apr 2025, Singapore, Singapore. ⟨hal-05441473⟩
92 Consultations
168 Téléchargements

Partager

  • More