The ability to leverage the power of a network of social contacts is important to get things done. However, as the number of contacts increases, people often find it difficult to maintain their contact network by using merely memory, and are frequently encompassed with questions like "who is that person, I met him in Tokyo last year". Existing contact tools make up for the shortage of unreliable human memory by storing contact information in the digital format, but laying much burden on users on manually inputting contact data. This paper, however, presents a social contact management system called SCM, which supports the auto-collection of rich contact data by exploring the aggregated power of pervasive sensing and Web intelligence techniques. Regarding that people often need to leverage several associated things (e.g., meeting location) to fetch other information about a contact (e.g., his name), we also develop an associative contact retrieval method. The effectiveness and runtime performance of our system is validated through a set of experiments