Recent tracking methods in professional team sports reach very high accuracy by tracking the ball and players. However, it remains difficult for these methods to perform accurate real-time tracking in amateur acquisition conditions where the vertical position or orientation of the camera is not controlled and cameras use heterogeneous sensors. This article presents a method for tracking interesting content in an amateur sport game by analyzing player displacements. Defining optical flow of the foreground in the image as the player motions, we propose a piecewise linear supervised learning model for predicting the camera global motion needed to follow the action.