In this work, we propose a new formulation of the objects modeling combining geometry and appearance; it is useful for detection and recognition. The object local appearance location is referenced with respect to an invariant which is a geometric landmark. The appearance (shape and texture) is a combination of Harris–Laplace descriptor and local binary pattern (LBP), all being described by the invariant local appearance model (ILAM). We use an improved variant of LBP traits at regions located by Harris–Laplace detector to encode local appearance. We applied the model to describe and learn object appearances (e.g., faces) and to recognize them. Given the extracted visual traits from a test image, ILAM model is carried out to predict the most similar features to the facial appearance: first, by estimating the highest facial probability and then in terms of LBP histogram-based measure, by computing the texture similarity. Finally, by a geometric calculation the invariant allows to locate an appearance in the image. We evaluate the model by testing it on different face images databases. The experiments show that the model results in high accuracy of detection and provides an acceptable tolerance to the appearance variability.