We simulate Lagrangian drift on the sea surface and investigate deep learning approaches to address the shortcomings of current model-based and Markovian approaches, particularly concerning error propagation and computational complexity. We present a novel deep learning framework, referred to as DriftNet, inspired by the Eulerian Fokker-Planck representation of Lagrangian dynamics. Through numerical experiments for simulated and real drift trajectories on the sea surface, we illustrate the effectiveness of DriftNet compared to existing state-of-the-art schemes. We also delve into the influence of diverse geophysical fields, whether derived from models or observations, used as inputs by DriftNet on drift simulation. Our objective is to assess the amount of dynamic information required to accurately simulate realistic trajectories.