In the current context of climate change, tree-crop combinations in agroforestry systems are suggested to mitigate water and heat stresses, particularly in semi-arid environments of the Mediterranean area. In this framework, a 3-year trial was conducted at the French National Research Institute for Agriculture, Food and the Environment (INRAE) in Mauguio (Southern France) in order to investigate the response of twenty-five durum wheat genotypes under a yearly pruned (AF) and a never-pruned alley olive orchard (AF+), in comparison with an open field without trees (control, C). The grain yield of wheat was markedly reduced in both the agroforestry systems AF (average -43%) and AF+ (-83%), according to the shading level. Among the yield components, the plant density at harvest was enhanced in AF (+22%) and AF+ (+3%), although with a significant reduction in the number of grains per spike (-37% in AF and -62% in AF+), and the number of spikes per plant (-32% in AF and -52% in AF+). The thousand-grain weight (TGW) and harvest index (HI) were slightly higher under moderate shade (AF; +12% vs. C) and severe shading (AF+; +6%). Plant biomass and spike size were significantly reduced in both agroforestry systems, while the flag leaf-spike distance (last internode) increased in AF. It was concluded that the moderate shading conditions of AF may create a sustainable agricultural system, and the wide intraspecific variability suggested a large scope for screening suitable genotypes, helping to produce ideotypes to implement agroforestry-oriented breeding programs.