Throughout tomato domestication, a large increase in fruit size was associated with a loss of dry matter and sugar contents. This study aims to dissect the contributions of genetic variation and the physiological processes underlying the relationships between fruit growth and the accumulation of dry matter and sugars. Fruit quality traits and physiological parameters were measured on 20 introgression lines derived from the introgression of Solanum chmielewskii into S. lycopersicum, under high (HL, unpruned trusses) and low (LL, trusses pruned to one fruit) fruit load conditions. Inter- and intra-genotypic correlations among traits were estimated and quantitative trait loci (QTL) for size, composition, and physiological traits were mapped. LL increased almost all traits, but the response of sugar content was genotype-dependent, involving either dilution effects or differences in carbon allocation to sugars. Genotype3fruit load interactions were significant for most traits and only 30% of the QTL were stable under both fruit loads. Many QTL for fresh weight and cell or seed numbers co-localized. Eleven clusters of QTL for fresh weight and dry matter or sugar content were detected, eight with opposite allele effects and three with negative effects. Two genotypic antagonistic relationships, between fresh weight and dry matter content and between cell number and cell size, were significant only under HL; the second could be interpreted as a competition for carbohydrates among cells. The role of cuticular conductance, fruit transpiration or cracking in the relationship between fruit fresh weight and composition was also emphasized at the genetic and physiological levels.