There are reasons to reconstruct a surface from a sparse cloud of 3D points estimated from an image sequence: to avoid computationally expensive dense stereo, e.g. for applications that do not need high level of details and have limited resources, or to initialize dense stereo in other cases. It is also interesting to enforce topology constraints (like manifoldness) for both surface regularization and applications. In this article, we improve by several ways a previous method that enforces the manifold constraint given a sparse point cloud. We enforce lowered genus, i.e. simplified topology, as a further regularization constraint for maximizing the visibility consistency encoded in a 3D Delaunay triangulation of the points. We also provide more efficient escapes from local extrema, an acceleration of the manifold test and more efficient removals of surface singularities. We experiment on a sparse point cloud reconstructed from videos, that are taken by a helmet-held omnidirectional multi-camera moving in an university campus.