Abstract Although mammalian vocalizations are predominantly harmonically structured, they can exhibit an acoustic complexity with nonlinear vocal sounds, including deterministic chaos and frequency jumps. Such sounds are normative events in mammalian vocalizations, and can be directly traceable to the nonlinear nature of vocal-fold dynamics underlying typical mammalian sound production. In this study, we give qualitative descriptions and quantitative analyses of nonlinearities in the song repertoire of humpback whales from the Ste Marie channel (Madagascar) to provide more insight into the potential communication functions and underlying production mechanisms of these features. A low-dimensional biomechanical modeling of the whale’s U-fold (vocal folds homolog) is used to relate specific vocal mechanisms to nonlinear vocal features. Recordings of living humpback whales were searched for occurrences of vocal nonlinearities (instabilities). Temporal distributions of nonlinearities were assessed within sound units, and between different songs. The anatomical production sources of vocal nonlinearities and the communication context of their occurrences in recordings are discussed. Our results show that vocal nonlinearities may be a communication strategy that conveys information about the whale’s body size and physical fitness, and thus may be an important component of humpback whale songs.