Some recent results showed that the renormalization group (RG) can be considered as a promising framework to address open issues in data analysis. In this work, we focus on one of these aspects, closely related to principal component analysis (PCA) for the case of large dimensional data sets with covariance having a nearly continuous spectrum. In this case, the distinction between ‘noise-like’ and ‘non-noise’ modes becomes arbitrary and an open challenge for standard methods. Observing that both RG and PCA search for simplification for systems involving many degrees of freedom, we aim to use the RG argument to clarify the turning point between noise and information modes. The analogy between coarse-graining renormalization and PCA has been investigated in Bradde and Bialek (2017 J. Stat. Phys. 167 462–75), from a perturbative framework, and the implementation with real sets of data by the same authors showed that the procedure may reflect more than a simple formal analogy. In particular, the separation of sampling noise modes may be controlled by a non-Gaussian fixed point, reminiscent of the behaviour of critical systems. In our analysis, we go beyond the perturbative framework using nonperturbative techniques to investigate non-Gaussian fixed points and propose a deeper formalism allowing us to go beyond power-law assumptions for explicit computations.