Towards Resilient Transformer-Encoders: Fault Injection and Hardware Agnostic Error Mitigation
Résumé
Transformer-based models are being deployed in safety-critical domains such as autonomous systems and healthcare, where reliability is important. Fault tolerance is crucial when a model's stability is impacted due to transient and permanent hardware faults. However, fault analysis and mitigation in these architectures remain limited. Thresholding, quantization, and a new method called Bit Mask can be used to mitigate the accuracy degradation caused by bit-flips in hidden states. To validate these methods, we simulated faults by injecting errors in the hidden states of the model. Both fine-grained and coarse-grained thresholding methods improved resilience with reduction in the metric drop from 41% to below 1% and the accuracy degradation for Bit Mask reduced to less than 2.5%. Our work also conducts an extensive fault injection campaign on Query, Key and Value weight matrices of transformer models revealing the Value weight matrix as most critical, particularly at the third exponent bit.
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| Licence |