Automatic image-based-modeling usually has two steps: Structure from Motion (SfM) and the estimation of a trian-gulated surface. The former provides camera poses and a sparse point cloud. The latter usually involves dense stereo. From the computational standpoint, it would be nice to avoid dense stereo and estimate the surface from the sparse cloud directly. Furthermore, it would be useful for online applications to update the surface while the camera is moving in the scene. This paper deals with both requirements: it introduces an incremental method which reconstructs a surface from a sparse cloud estimated by incremental SfM. The context is new and difficult since we ensure the resulting surface to be manifold at all times. The manifold property is important since it is needed by differential operators involved in surface refinements. We have experimented with a hand-held omnidirectional camera moving in a city.