Hole shrinkage is a common phenomenon in drilling difficult-to-cut materials like Ti6Al4V due to their poor thermal properties and high elasticity, which can lead to increase in tool wear and decrease in surface integrity. In this study, an in-depth analysis of hole shrinkage mechanism is carried out through a hybrid modelling approach for both dry and cryogenic cutting conditions. The plastic deformation induced by chip formation and tool-workpiece interaction is treated as equivalent thermomechanical loads, and heat convection conditions are described along tool path in order to perform details in heat transfer process for both cases. Quantitative analysis is presented through numerical simulation and experimental data of temperature and deformation along hole contour to analyze deformation components in order to reveal the hole shrinkage mechanism. Additional interactions between cutting tool and workpiece material are induced by recovery of elastoplastic deformation, and plastic portion is a more devastating factor in tool wear and surface damage induced by hole shrinkage. This study presents an in-depth and fundamental understanding of the hole shrinkage mechanism through a hybrid modelling approach, which can characterize heat transfer process during machining for both dry and cryogenic conditions, and simulation of this fully coupled thermomechanical cutting process with supply of coolants was rarely reported in previous research. The results show that plastic deformation induced hole shrinkage can enhance the interaction between workpiece material and cutting tool, and cryogenic assistance presents a good performance in restricting this kind of phenomenon. The related results could be used to optimize strategies of eliminating hole shrinkage with cryogenic assistance in industrial applications.