Lagrangian drift simulations are widely used in operational oceanography. Among others, simulated trajectories at the sea surface can be used in plastic drift and oil spills diffusivity prediction as well as in Search and Rescue missions. Whereas state-of-the-art operational schemes generally rely on physics-informed drift models, machine learning approaches arise as appealing alternatives, especially regarding the potential impact of fine- scale sea surface dynamics on Lagrangian trajectories and the limited ability of operational products to reconstruct these sea surface fine-scale dynamics. Here, we present a novel deep learning scheme for the simulation of Lagrangian drift at sea surface. Our network, called DriftNet, is a fully convolutional generative model including a convolutional LSTM unit. It uses as inputs the sea surface ocean current conditions around the initial position of the simulated trajectory. Inspired by Fokker-Planck equation for Lagrangian dynamics, DriftNet mimics an advection scheme where the trainable spatially-explicit field-based representation of the particle’s distribution is advected by the sea surface currents. We run numerical experiments for a case-study area of California using the ocean surface currents of the global Copernicus Marine service reanalysis products as input for our model. DriftNet outperforms classic Deep Learning approaches for the simulation of multivariate time series and trajectory data, including LSTM-based architectures. Beyond its simulation performance, we showcase how DriftNet can further be used as a novel inversion scheme to provide reanalysis sea surface current corrections from real drifters’ trajectories data.