Physics-based Digital Twins, particularly those using the finite element method to solve the underlying partial differential equation, accurately simulate organ behaviors but are computationally intensive, especially for hyper-elastic tissues. Recently, approaches have leveraged neural-network-based surrogate models to accelerate computation time. However, these models are limited by the accurate knowledge of patient-specific characteristics, such as material properties and boundary conditions, at training time. This paper introduces a novel methodology for patient-specific characteristics estimation from live observations during medical interventions. To retain the benefits of neural network-based surrogate models, we propose a hypernetwork architecture that conditions the surrogate models on patient-specific characteristics, thus maintaining accuracy over a predefined distribution of these characteristics. Using the trained network, we perform a gradient-based optimization process to determine the patient characteristics given an intraoperative observation. We demonstrate the flexibility and efficiency of our approach through experiments with varying geometries, complex physics laws, and various patient characteristics.