This paper deals with the design of a generic visual tracking algorithm suitable for a large class of camera (single viewpoint sensors). It is based on the estimation of the relationship between observations and motion on the sphere. This is efficiently achieved using a kernel-based regression function on a generic linearly-weighted sum of non-linear basis functions. We also present two set of experiments. The first one shows the efficiency of our algorithm through the tracking in video sequences acquired with three types of cameras (conventional, dioptric-fisheye and catadioptric). The real-time performances will be shown by tracking one or several planes. The second set of experiments presents an application of our tracking algorithm to visual servoing with a fisheye camera.