Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores - Archive ouverte HAL Access content directly
Journal Articles SIAM Journal on Scientific Computing Year : 2020

Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores

Abstract

Computing units that carry out a fused multiply-add (FMA) operation with matrix arguments, referred to as tensor units by some vendors, have great potential for use in scientific computing. However, these units are inherently mixed precision and existing rounding error analyses do not support them. We consider a mixed precision block FMA that generalizes both the usual scalar FMA and existing tensor units. We describe how to exploit such a block FMA in the numerical linear algebra kernels of matrix multiplication and LU factorization and give detailed rounding error analyses of both kernels. An important application is to GMRES-based iterative refinement with block FMAs, for which our analysis provides new insight. Our framework is applicable to the tensor core units in the NVIDIA Volta and Turing GPUs. For these we compare matrix multiplication and LU factorization with TC16 and TC32 forms of FMA, which differ in the precision used for the output of the tensor cores. Our experiments on an NVDIA V100 GPU confirm the predictions of the analysis that the TC32 variant is much more accurate than the TC16 one, and they show that the accuracy boost is obtained with almost no performance loss.
Fichier principal
Vignette du fichier
BlockFMA.pdf (390.05 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-02491076 , version 1 (25-02-2020)
hal-02491076 , version 2 (28-05-2020)

Identifiers

Cite

Pierre Blanchard, Nicholas J Higham, Florent Lopez, Théo Mary, Srikara Pranesh. Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores. SIAM Journal on Scientific Computing, 2020, 42 (3), pp.C124-C141. ⟨10.1137/19M1289546⟩. ⟨hal-02491076v2⟩
119 View
586 Download

Altmetric

Share

Gmail Facebook X LinkedIn More