In this paper, we contribute to solve the simultaneous problems of face detection and gender classification from any viewpoint. We use an invariant model for accurate face localization based on a combination of appearance and geometry. A probabilistic matching of visual traits allows to classify the gender of face even when pose changes. We deal with the local invariant features whose performances have already been proved. Each facial feature retained in the detection step will be weighted by a probability to be male or female. This feature contributes to determine the gender of the face. We evaluate our model by testing it in experiments on different databases. The experimental results show that the face model performs well to detect face and gives a good gender recognition rate in the presence of viewpoint changes and facial appearance variability.