This paper designs an Image-based Ensemble Kalman Filter (IEnKF), whose components are defined only from image properties, to estimate motion on image sequences. The key elements of this filter are, first, the construction of the initial ensemble, and second, the propagation in time of this ensemble on the studied temporal interval. Both are analyzed in the paper and their impact on results is discussed with synthetic and real data experiments. The initial ensemble is obtained by adding a Gaussian vector field to an estimate of motion on the first two frames. The standard deviation of this normal law is computed from motion results given by a set of optical flow methods of the literature. It describes the uncertainty on the motion value at initial date. The propagation in time of the ensemble members relies on the following evolution laws: transport by velocity of the image brightness function and Euler equations for the motion function. Shrinking of the ensemble is avoided thanks to a localization method and the use of observation ensembles, both techniques being defined from image characteristics. This Image-based Ensemble Kalman Filter is quantified on synthetic experiments and applied on traffic and meteorological images.