Restoring images degraded by spatially varying blur is a problem encountered in many disciplines such as astrophysics, computer vision or biomedical imaging. One of the main challenges to perform this task is to design efficient numerical algorithms to approximate integral operators. We review the main approaches developped so far and detail their pros and cons. We then analyze the numerical complexity of the mainstream approach based on piecewise convolutions. We show that this method provides an $\epsilon$-approximation of the matrix-vector product in $\mathcal{O}\left(N^d \log(N) \epsilon^{-d}\right)$ operations where $N^d$ is the number of pixels of a $d$-dimensional image. Moreover, we show that this bound cannot be improved even if further assumptions on the kernel regularity are made. We then introduce a new method based on a sparse approximation of the blurring operator in the wavelet domain. This method requires $\mathcal{O}\left(N^d \epsilon^{-d/M}\right)$ operations to provide $\epsilon$-approximations, where $M\geq 1$ is a scalar describing the regularity of the blur kernel. We then propose variants to further improve the method by exploiting the fact that both images and operators are sparse in the same wavelet basis. We finish by numerical experiments to illustrate the practical efficiency of the proposed algorithms.