This paper presents a general audio classification approach inspired by our modest knowledge about the human perception of sound. Simple psychoacoustic experiments show that the relation between short term spectral features has a great impact on the human audio classification performance. For instance, short term spectral features extracted from speech sound can be perceived as non-speech sounds if organized in a special way in time. We have developed the idea of incorporating several consecutive spectral features when modelling the audio signal in relatively long term time windows. The modelling scheme that we propose, piecewise Gaussian modelling (PGM), was combined with a neural network to develop a general audio classifier. The classifier was evaluated on the problems of speech/music classification. male/female classification and special events detection in sports videos. The good classification accuracy obtained by the classifier suggests us to continue the research in order to improve the model and to closely combine it to some well-known psychoacoustic experimental results.