In this paper, we address a problem of gender classification of faces taken from arbitrary viewpoints. We use a face model for accurate face localization based on a combination of appearance and geometry. A probabilistic matching of particular traits on face allows to classify the gender of face even in case of important pose changes. We deal with the local invariant features whose performances have already been proved. Each facial feature retained in the detection step will be weighted by a probability to be male or female. Such feature contributes to determine the gender associated to a given face. We evaluate the model by testing it simultaneously in face localization and gender classification experiments on PIE, FERET and CMU-Profiles databases. The experimental results show that the probabilistic invariant model performs well to detect faces and gives a rate of 92.1% of accurate gender classification in the presence of viewpoint changes and large appearance variability of faces.