Let {F_n} be a normalized sequence of random variables in some fixed Wiener chaos associated with a general Gaussian field, and assume that E[F_n^4] --> E[N^4]=3, where N is a standard Gaussian random variable. Our main result is the following general bound: there exist two finite constants c,C>0 such that, for n sufficiently large, c max(|E[F_n^3]|, E[F_n^4]-3) < d(F_n,N) < C max(|E[F_n^3]|, E[F_n^4]-3), where d(F_n,N) = sup |E[h(F_n)] - E[h(N)]|, and h runs over the class of all real functions with a second derivative bounded by 1. This shows that the deterministic sequence max(|E[F_n^3]|, E[F_n^4]-3) completely characterizes the rate of convergence (with respect to smooth distances) in CLTs involving chaotic random variables. These results are used to determine optimal rates of convergence in the Breuer-Major central limit theorem, with specific emphasis on fractional Gaussian noise.