On the Fault Sensitivity of Natural Language Embeddings Computation
Résumé
Transformer-based text embedding models have completely changed the landscape of natural language processing domain resulting in enhanced semantic understandings. They are pivotal to achieve high accuracies for various tasks like classification, semantic similarity, and are integral part of Retrieval Augmented Generation (RAG) pipelines (in conjunction with Large Language Models (LLMs)). Embeddings on natural language entries can be computed at the edge, thanks to the inherent compression they provide before communication to the cloud where LLMs are usually executed, thus their computation is critical in terms of accuracy, power and reliability. This work focuses on the latter, by conducting extensive error sensitivity analysis on the text embedding transformer models, targeting different applications using state-of-the-art models. Interestingly, the error injection campaign identifies that mainly a portion of the self-attention layer, specifically the Value tensor (V), is very sensitive to single bit-flips. These results pave the way for a more effective error tolerance approach in future transformer-based embeddings and possibly other self-attention based models.
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| Licence |