DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables

Darshan C Ganji
  • Fonction : Auteur
  • PersonId : 1250345
Saad Ashfaq
  • Fonction : Auteur
  • PersonId : 1250346
Ehsan Saaboori
  • Fonction : Auteur
  • PersonId : 1250347
Sudhakar Sah
  • Fonction : Auteur
  • PersonId : 1250348
Saptarshi Mitra
  • Fonction : Auteur
  • PersonId : 1250349
Mohammadhossein Askarihemmat
  • Fonction : Auteur
  • PersonId : 1250350
Alexander Hoffmann
  • Fonction : Auteur
  • PersonId : 928843
Ahmed Hassanien
  • Fonction : Auteur
  • PersonId : 1250351

Résumé

A lot of recent progress has been made in ultra lowbit quantization, promising significant improvements in latency, memory footprint and energy consumption on edge devices. Quantization methods such as Learned Step Size Quantization can achieve model accuracy that is comparable to full-precision floating-point baselines even with subbyte quantization. However, it is extremely challenging to deploy these ultra low-bit quantized models on mainstream CPU devices because commodity SIMD (Single Instruction, Multiple Data) hardware typically supports no less than 8-bit precision. To overcome this limitation, we propose DeepGEMM, a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. The proposed method precomputes all possible products of weights and activations, stores them in a lookup table, and efficiently accesses them at inference time to avoid costly multiply-accumulate operations. Our 2-bit implementation outperforms corresponding 8-bit integer kernels in the QNNPACK framework by up to 1.74× on x86 platforms.
Fichier principal
Vignette du fichier
2304.09049.pdf (412.79 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04082069 , version 1 (26-04-2023)

Identifiants

Citer

Darshan C Ganji, Saad Ashfaq, Ehsan Saaboori, Sudhakar Sah, Saptarshi Mitra, et al.. DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables. CVPRW 2023: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Jun 2023, Vancouver, Canada. ⟨10.1109/CVPRW59228.2023.00491⟩. ⟨hal-04082069⟩
21 Consultations
92 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More