We propose a new method to efficiently price swap rates derivatives under the LIBOR Market Model with Stochastic Volatility and Displaced Diffusion (DDSVLMM). This method uses polynomial processes combined with Gram-Charlier expansion techniques. The standard pricing method for this model relies on dynamics freezing to recover an Heston-type model for which analytical formulas are available. This approach is time consuming and efficient approximations based on Gram-Charlier expansions have been recently proposed. In this article, we first discuss the fact that for a class of stochastic volatility model, including the Heston one, the classical sufficient condition ensuring the convergence of the Gram-Charlier series can not be satisfied. Then, we propose an approximating model based on Jacobi process for which we can prove the stability of the Gram-Charlier expansion. For this approximation, we have been able to prove a strong convergence toward the original model; moreover, we give an estimate of the convergence rate. We also prove a new result on the convergence of the Gram-Charlier series when the volatility factor is not bounded from below. We finally illustrate our convergence results with numerical examples.