This article extends previous work by the authors on the single- and multi-domain time-harmonic elastodynamic multi-level fast multipole BEM formulations to the case of weakly dissipative viscoelastic media. The underlying boundary integral equation and fast multipole formulations are formally identical to that of elastodynamics, except that the wavenumbers are complex-valued due to attenuation. Attention is focused on evaluating the multipole decomposition of the viscoelastodynamic fundamental solution. A damping-dependent modification of the selection rule for the multipole truncation parameter, required by the presence of complex wavenumbers, is proposed. It is empirically adjusted so as to maintain a constant accuracy over the damping range of interest in the approximation of the fundamental solution, and validated on numerical tests focusing on the evaluation of the latter. The proposed modification is then assessed on 3D single-region and multi-region visco-elastodynamic examples for which exact solutions are known. Finally, the multi-region formulation is applied to the problem of a wave propagating in a semi-infinite medium with a lossy semi-spherical inclusion (seismic wave in alluvial basin). These examples involve problem sizes of up to about $3\,10^{5}$ boundary unknowns.