In this paper, we introduce a new class of processes which are diffusions with jumps, where the jumps are driven by a multivariate linear Hawkes process, and study their long-time behavior. In the case of exponential memory kernels for the underlying Hawkes process, we establish conditions for the positive Harris recurrence of the couple (X, λ), where X denotes the diffusion process and λ the stochastic intensity of the driving Hawkes. As a direct consequence of the Harris recurrence, we obtain the ergodic theorem for X. Furthermore, we provide sufficient conditions under which the process is exponentially β−mixing. This paper is the foundation for a second paper [11] in which we carry a statistical study of diffusions driven by Hawkes jumps, with a view towards applications in neuroscience.