This paper presents a system for capturing the deformations of soft objects undergoing elastic deformations and contacts with their environment, using image and point cloud data provided by an RGB-D sensor. We improve upon previous works by integrating environment constraints in the frame-by-frame registration process. The approach combines a physics-based elastic model of the considered objects, computed in real-time using an optimized Finite Element Method (FEM), which is driven by surface constraints on the objects. Additional forces, such as gravity are added. A case study in open surgery on the liver is here described. Yet in this case a major improvement in the accuracy of the registration is provided by the integration of anatomical shape constraints, which are naturally hidden from the RGB-D camera, and that we account for through a registration with the pre-operative CT data. With a comparative study, we demonstrate the relevance of our method in a real world application mimicking an open surgery scenario where the liver has to be tracked to provide an augmented reality view.