The diffusion magnetic resonance imaging signal arising from biological tissues can be numerically simulated by solving the Bloch-Torrey partial differential equation. Numerical simulations can facilitate the investigation of the relationship between the diffusion MRI signals and cellular structures. With the rapid advance of available computing power, the diffusion MRI community has begun to employ numerical simulations for model formulation and validation, as well as for imaging sequence optimization. Existing simulation frameworks use the finite difference method, the finite element method, or the Matrix Formalism method to solve the Bloch-Torrey partial differential equation. We propose a new method based on the efficient evaluation of layer potentials. In this paper, the mathematical framework and the numerical implementation of the new method are described. We demonstrate the convergence of our method via numerical experiments and analyze the errors linked to various model and simulation parameters. Since our method provides a Fourier-type representation of the diffusion MRI signal, it can potentially facilitate new physical and biological signal interpretations in the future.