Communication Dans Un Congrès Année : 2026

Towards Resilient Transformer-Encoders: Fault Injection and Hardware Agnostic Error Mitigation

Résumé

Transformer-based models are being deployed in safety-critical domains such as autonomous systems and healthcare, where reliability is important. Fault tolerance is crucial when a model's stability is impacted due to transient and permanent hardware faults. However, fault analysis and mitigation in these architectures remain limited. Thresholding, quantization, and a new method called Bit Mask can be used to mitigate the accuracy degradation caused by bit-flips in hidden states. To validate these methods, we simulated faults by injecting errors in the hidden states of the model. Both fine-grained and coarse-grained thresholding methods improved resilience with reduction in the metric drop from 41% to below 1% and the accuracy degradation for Bit Mask reduced to less than 2.5%. Our work also conducts an extensive fault injection campaign on Query, Key and Value weight matrices of transformer models revealing the Value weight matrix as most critical, particularly at the third exponent bit.

Fichier principal
Vignette du fichier
Towards_Resilient_Transformer_Encoders__Fault_Injection_and_Hardware_Agnostic_Error_Mitigation.pdf (790.72 Ko) Télécharger le fichier

Dates et versions

hal-05574966 , version 1 (31-03-2026)

Licence

Identifiants

  • HAL Id : hal-05574966 , version 1

Citer

Vishal Thomas, Ussama Zahid, Giulio Gambardella, Sumeet Sapkal, Haralampos-G. Stratigopoulos, et al.. Towards Resilient Transformer-Encoders: Fault Injection and Hardware Agnostic Error Mitigation. 31st IEEE European Test Symposium, May 2026, Chania, Greece. ⟨hal-05574966⟩
139 Consultations
62 Téléchargements

Partager

  • More