Long-term monitoring of human vital signs like respiration and heartbeat is crucial for the early detection of diverse diseases and overall health monitoring. Contact-free vital signs monitoring using wireless signals, particularly mmWave-based methods, has gained attention due to its sensitivity and privacy-preserving benefits. However, we observe that even minor human movements could lead to significant mutations in the signal-to-noise ratio (SNR) of the wireless signal, which cannot be explained by the commonly used model that represents the human chest as a single reflection point. These fluctuations challenge the robustness of heart rate and heart rate variability (HRV) monitoring due to the vulnerability of faint heartbeats to noise interference. To tackle this, we introduce a multi-point reflection model to understand the underlying causes of SNR fluctuations and propose a frequency diversity based algorithm to enhance sensing SNR. Our solution, Robust-Vital, was rigorously evaluated using commercial mmWave radar systems and demonstrated superior performance on long-term heart rate and heart rate variability tracking in a user study with 12 participants.