In this paper, we are interested in a diffusion process based on a gradient descent. The process is non Markov and has a memory term which is built as a weighted average of the drift term all along the past of the trajectory. For this type of diffusion, we study the long time behaviour of the process in terms of the memory. We exhibit some conditions for the long-time stability of the dynamical system and then provide, when stable, some convergence properties of the occupation measures and of the marginal distribution, to the associated steady regimes. When the memory is too long, we show that in general, the dynamical system has a tendency to explode, and in the particular gaussian case, we explicitly obtain the rate of divergence.