Passive tracking plays a fundamental role in numerous applications such as elderly care, security surveillance, and smart home. To utilize ubiquitous WiFi signals for passive tracking, the Doppler speed extracted from WiFi CSI (Channel State Information) is the key information. Despite the progress made, existing approaches still require a large number of samples to achieve accurate Doppler speed estimation. To enable WiFi sensing with minimum amount of interference on WiFi communication, accurate Doppler speed estimation with fewer CSI samples is crucial. To achieve this, we build a passive WiFi tracking system which employs a novel CSI difference paradigm instead of CSI for Doppler speed estimation. In this paper, we provide the first deep dive into the potential of CSI difference for fine-grained Doppler speed estimation. Theoretically, our new design allows us to estimate Doppler speed with just three samples. While conventional methods only adopt phase information for Doppler estimation, we creatively fuse both phase and amplitude information to improve Doppler estimation accuracy. Extensive experiments show that our solution outperforms the state-of-the-art approaches, achieving higher accuracy with fewer CSI samples. Based on this proposed WiFi-CSI difference paradigm, we build a prototype passive tracking system which can accurately track a person with a median error lower than 34 cm, achieving similar accuracy compared to the state-of-the-art systems, while significantly reducing the required number of samples to only 5%.