Second-order descriptors play an increasingly important role in many signal and image processing applications including for example remote sensing and medical imaging. In order to develop machine learning models adapted to this kind of descriptors, different probabilistic models based on the Gaussian assumption have been proposed. This includes intrinsic models on the manifold of symmetric positive definite (SPD) matrices such as the Riemannian Gaussian distribution, but also conventional Gaussian models defined on a tangent plane at a reference point. Even if the former is defined on the manifold, it suffers from a practical point of view (no close form expression for the normalization factor, scalar dispersion parameter). The Gaussian model on the tangent plane does not have these drawbacks but is limited by a fixed reference point which might lead to some distortions. To overcome these difficulties, we propose to define a Gaussian model on the tangent plane where the reference point is learned. We found that the maximum likelihood estimator of this reference point is the Karcher/Fréchet mean. Based on this, we introduce a Gaussian mixture model (GMM) with multiple reference points and derive the maximum likelihood estimator. Experimental results on synthetic dataset show that the proposed approach allows to limit the distortion while having an anisotropic dispersion matrix. Finally, an experiment on remote sensing image scene classification is performed to illustrate the potential of the proposed GMM model with a Fisher vector encoding of second-order descriptors computed on the feature maps of a convolutional neural network.