Internet telephony permit callers to manage self-asserted profiles without any subscription contract nor identification proof. These cost-free services have attracted many telemarketers and spammers who generate unsolicited nuisance calls. Upon detection, they simply rejoin the network with a new identity to continue their malicious activities. Nuisance calls are highly disruptive when compared to email and social spam. They not only include annoying telemarketing calls but also contain scam and voice phishing which involves security risk for subscribers. Therefore, it remains a major challenge for Internet telephony providers to detect and avoid nuisance calls efficiently. In this paper, we present a new approach that uses caller reputation to detect different kinds of nuisance calls generated in the network. The reputation is computed in a hybrid manner by extracting information from call data records and using recommendations from reliable communicating participants. The behavior of the caller is assessed by extracting call features such as call-rate, call duration, and call density. Long term and short term reputations are computed to quickly detect the changing behavior of callers. Furthermore, our approach involves an efficient mechanism to combat whitewashing attacks performed by malicious callers to continue generating nuisance calls in the network. We conduct simulations to compute the performance of our proposed model. The experiments conclude that the proposed reputation model is an effective method to detect different types of nuisance calls while avoiding false detection of legitimate calls.