While it is very hard to achieve automatic sports competition key moments detection only based on visual analysis, we propose in this paper automatic highlights detection based on an audio classifier. The audio classifier is based on a new modeling technique of the audio spectrum called Piecewise Gaussian Modeling (PGM) and Neural Networks. The proposed approach was evaluated on soccer and tennis videos, though our technique has no restriction on the sports’ type. It is shown that audio-based highlights detection can be effective for tennis segmentation since 97.5 % of end-of-serves were correctly classified. Goals can be detected in soccer videos using audio analysis as well. An intelligent sports-videos player is proposed based on the audio analysis permitting the user to navigate through key moments in a sports video.