In this paper we propose a novel method to recover the 3D shape of the knee and of the prosthesis for the patients that have undergone total knee replacement surgery. We address this using a set of 2D X-Ray images of the knee taken from different viewpoints around it. The problem is then casted to a multiview stereo reconstruction problem and we consider an energy-minimization approach to recover the bone structure. This function is designed to recover a 3D surface which minimizes image and shape-based terms. The cost related to data is defined through the projections of the 3D surface onto 2D image planes while the prior-knowledge is introduced using a 3D model of the knee. The use of Markov Random Fileds as well as recent advances in discrete optimization and in particular the state-of-the-art Fast PD optimization algorithm is considered towards optimal reconstruction. Promising results demonstrate the potentials of our method.