Background—Adults with Down’s syndrome (DS) present lower physical fitness associated with heightened sedentary behaviors and motor skills impairments. Their etiologies and determinants seem to be heterogeneous. This study aims to evaluate physical fitness in adults with DS and to identify specific physical fitness profiles depending on gender and physical activity levels. Methods—Forty adults with DS (16 women, 24 men, 29.7 ± 7.5 years) performed six tests from the EUROFIT Battery and Motor Assessment Battery for Children (MAB-C). Their maximal aerobic capacity was assessed using an incremental treadmill test to assess (VO2peak). Ecological, physical activity, and sedentary levels were evaluated subjectively (Global Physical Activity Questionnaire) and objectively using an Actigraph GT9X® accelerometer over a seven-day period. Results—VO2peak and isometric strength were significantly lower for women (p < 0.01), whereas men had significantly lower flexibility than women (p < 0.05). Using a principal component analysis and an agglomerative hierarchical analysis, we identified three clusters. Cluster 1 (n = 14; 50% men; Body Mass index = 28.3 ± 4.3) was characterized by significantly poorer physical fitness variables (VO2peak (p < 0.01), strength (p < 0.01) and balance (p < 0.05)) compared to Clusters 2 and 3. Cluster 2 (n = 19; 58% men; Body Mass index = 22.9 ± 2.0) and Cluster 3 (n = 19; 58% men; BMI = 22.9 ± 1.9) were characterized by subjects with comparable physical fitness profiles, except for the balance capacities, which were significantly lower in Cluster 3 (p < 0.05). Conclusions—DS subjects exhibited high heterogeneity in terms of physical fitness, PA, and sedentary levels, with a significant gender effect. The present findings are important to identify subjects at higher risk of sedentary behaviors and impaired motor capacities to develop personalized PA programs.