The application of network compression methods, combining tensor decompositions and pruning, has demonstrated significant potential in harnessing the benefits of both strategies. This research introduces NORTON (hybrid Network cOmpression thRough Tensor decompOsitions and pruNing), a novel hybrid approach for network compression. The key innovation lies in the introduction of filter decomposition, enhancing the detailed breakdown of the network while preserving the multidimensional properties of filters. Our method incorporates a structured pruning approach, seamlessly integrating the decomposed model. Through a comprehensive set of experiments across various architectures, benchmark datasets, and representative vision tasks, the efficacy of our approach is highlighted.