Side-information based exponential discriminant analysis (SIEDA) is more efficient than side-information based linear discriminant analysis (SILDA) in computing the discriminant vectors because it maximizes the Fisher criterion function. In this paper, we develop a novel criterion, named side-information based weighted exponential discriminant analysis (SIWEDA), that is based on the classical SIEDA method. We reformulate and generalize the classical Fisher criterion function in order to maximize it, with the property to pull as close as possible the intra-class samples (within-class samples), and push and repulse away as far as possible the inter-class samples (between-class samples). Thus, SIWEDA selects the eigenvalues of high significance and eliminate those with less discriminative information. To reduce the feature vector dimensionality and lighten the class intra-variability, we use SIWEDA and within class covariance normalization (WCCN) using the proposed statistical binarized image features (StatBIF). Moreover, we use score fusion strategy to extract the complementarity of different weighting scales of our StatBIF descriptor. We conducted experiments to evaluate the performance of the proposed method under unconstrained environment, using five datasets namely LFW, YTF, Cornell KinFace, UB KinFace and TSKinFace datasets, in the context of matching faces and kinship verification in the wild conditions. The experiments showed that the proposed approach outperforms the current state of the art. Very interestingly, our approach showed superior performance compared to methods based on deep metric learning.