We consider the transfer operators of non-uniformly expanding maps for potentials of various regularity, and show that a specific property of potentials (" flatness ") implies a Ruelle-Perron-Frobenius Theorem and a decay of the transfer operator of the same speed than entailed by the constant potential. The method relies neither on Markov partitions nor on inducing, but on functional analysis and duality, through the simplest principles of optimal transportation. As an application, we notably show that for any map of the circle which is expanding outside an arbitrarily flat neutral point, the set of Hölder potentials exhibiting a spectral gap is dense in the uniform topology. The method applies in a variety of situation, including Pomeau-Manneville maps with regular enough potentials, or uniformly expanding maps of low regularity with their natural potential; we also recover in a united fashion variants of several previous results.