This paper presents a tracking method to detect and track independently moving targets, attempting to traverse the railway, in monocular camera sequences. This method is capable of tracking the maximum number of pixels belonging to an object. The method starts by detecting and separating moving objects due to background subtraction and an energy vector-based clustering. Next, the method performs the step of tracking locally. Tracking starts by generating initial optical flow of all object pixels by propagating the optical flow of Harris corner points (calculated by Lucas\textendashKanade technique) using normal distribution. An iterative procedure, including Kalman filtering with adaptive parameters, color intensity difference-based optimization, and validation constraints, is then implemented to reach precise and robust optical flow estimation for the majority of the pixels of the tracked objects. Different experimental results are presented, evaluated, and discussed to show the effectiveness of the method of tracking objects that may move in complex and overlapping trajectories.