Problems with face recognition are frequent in older adults. However, the mechanisms involved have only been partially discovered. In particular, it is unknown to what extent these problems may be related to changes in configural face processing. Here, we investigated the face inversion effect (FIE) together with the ability to detect modifications in the vertical or horizontal second-order relations between facial features. We used a same/different unfamiliar face discrimination task with 33 young and 33 older adults. The results showed dissociations in the performances of older versus younger adults. There was a lack of inversion effect during the recognition of original faces by older adults. However, for modified faces, older adults showed a pattern of performance similar to that of young participants, with preserved FIE for vertically modified faces and no detectable FIE for horizontally modified faces. Most importantly, the detection of vertical modifications was preserved in older relative to young adults whereas the detection of horizontal modifications was markedly diminished. We conclude that age has dissociable effects on configural face-encoding processes, with a relative preservation of vertical compared to horizontal second-order relations processing. These results help to understand some divergent results in the literature and may explain the spared familiar face identification abilities in the daily lives of older adults.