Ocean eddies, as a ubiquitous phenomenon of the global ocean, are extremely important for ocean energy and material exchanges. Therefore, efficient eddy detection and tracking are crucial for advancing our understanding of ocean dynamics. This work presents a framework for automatic ocean eddy detection and tracking by leveraging state-of-the-art machine learning algorithms. First, we propose a new convolutional neural network model for multieddies detection. This model is capable of extracting accurate boundary information of eddies and fitting the gap between semantic context and sea surface height (SSH). Second, a tracking algorithm is designed to track eddies lasting a number of days and provide visualization of the dynamical processes governing eddies’ movements. Finally, we have made our data set publicly available, which is named SCSE-Eddy and can be used as a benchmark to evaluate the performances of artificial intelligence (AI)-based eddy detection methods. The data set covers daily remotely sensed SSH data located in the South China Sea and its eastern sea areas over a period of 15 years. The experimental results show that our methods achieve promising performances compared to existing approaches, especially for the eddies with indistinct geographical border. We believe that this work opens a new avenue for oceanographers to better discover and understand the physical properties of ocean eddies.