The creation of novel views using pre-stored images or image-based rendering has many potential applications, such as visual simulation, virtual reality and telepresence, for which traditional computer graphics based on geometric modeling would be unsatisfactory particularly with very complex 3D scenes. This paper presents a new image-based rendering system that tackles the two most difficults problems of image-based modeling: pixel matching and visibility handling. We first introduce the joint view triangulation (JVT), a novel representation for pairs of images that handles the visibility and occlusion problems created by the parallaxes between the images. The joint view triangulation is build from matched planar patches regularized by local smooth constraints encoded by plane homographies. Then, we introduce an incremental edge-constrained construction algorithm. Finally, we present a pseudo-painter's rendering algorithm for the joint view triangulation and demonstrate the performance of these methods experimentally.