Highlights • The signal statistics in 50 MHz high frequency ultrasound images of the dermis are studied accurately layer by layer. • An automatic, accurate segmentation method is developed, which is highly resistant to artifacts thanks to the use of elaborate non-linear filters and a multiple loss level-set segmentation algorithm. • Based on fits to a Nakagami law, a score is computed for characterizing skin photo aging. • The joint segmentation and characterization algorithm is designed to be applied to 2D and 3D images. Abstract We propose a novel joint segmentation and characterization algorithm for the assessment of skin aging using 50 MHz high-frequency ultrasound images. The proposed segmentation method allows a fine determination of the envelope signal's statistics in the dermis as a function of depth. The sequence of statistical estimates obtained is then combined into a single aging score. The segmentation is based on tailored recursive non-linear filters. The epidermis and the dermis are jointly segmented with a non-parametric active contour combining a texture criterion, an epidermis indicator map and the geometric constraint of horizontal continuity. The algorithm is designed to apply to 2D and 3D images as well. We evaluated skin photo-aging on ultrasound images with an experimental study on a cohort of 76 women separated into 2 groups of different ages. Two aging scores are computed from the images: local dermal contrast and skin roughness. We show that these scores are much better at identifying the two groups (p-value ) than the previously used indicator (p-value 0.046). Moreover, we find that a combined score more reliably evaluates skin photo-aging, with 84% success, than a scoring of the ultrasound images by 4 experts.