Several methods based on Total Variation (TV) have been proposed for Hyperspectral Image (HSI) denoising. However, the TV terms of these methods just use various $l_1$ norms and penalize image gradient magnitudes, having a negative influence on the preprocessing of HSI denoising and further HSI classification task. In this paper, a novel $l_0$ Total Variation ($l_0$ TV) is first introduced and analyzed for the HSI noise removal framework to preserve more information for classification. We propose a novel Tensor low-rank constraint and $l_0$ Total Variation (TLR-$l_0$ TV) model in this paper. $l_0$ TV directly controls the number of non-zero gradients and focuses on recovering the sharp image edges. The spectral-spatial information among all bands is exploited uniformly for removing mixed noise, which facilitates the subsequent classification after denoising. Including the Weighted Sum of Weighted Nuclear Norm (WSWNN) and the Weighted Sum of Weighted Tensor Nuclear Norm (WSWTNN), we propose two TLR-$l_0$ TV-based algorithms, namely WSWNN-$l_0$ TV and WSWTNN-$l_0$ TV. The Alternating Direction Method of Multipliers (ADMM) and the Augmented Lagrange Multiplier (ALM) are employed to solve the $l_0$ TV model and TLR-$l_0$ TV model, respectively. In both simulated and real data, the proposed models achieve superior performances in mixed noise removal of HSI. Especially, HSI classification accuracy is improved more effectively after denoising by the proposed TLR-$l_0$ TV method.