In social insects, within-colony worker diversity has long been thought to improve colony fitness. Recent studies start to question this assertion in species with moderate diversity. However, demonstrating an absence of effect is difficult as this absence could be due to the measurement of only few life history traits or to artificial conditions. In order to circumvent these limitations, we experimentally decreased worker size diversity within colonies of the ant Temnothorax nylanderi, with or without affecting mean worker size and we reintroduced them in the field for four months (spring and early summer). We then thoroughly measured their fitness based on survival, growth and reproductive success. Our results show that our manipulation did not affect colony fitness. In addition, colonies did not restore diversity to its initial level, further suggesting that worker size diversity is not a key parameter to them. We found the classically observed positive relationship between colony size, colony growth and reproductive success. Overall, our results confirm that worker size diversity within colony is not necessarily adaptive in species where it is moderate. We discuss the alternative mechanisms that could explain the evolutionary persistence of moderate worker size diversity. Significance statement Organisms that live in groups can greatly benefit from the emergence of novel group-level traits. For instance, social insects show significant variability in worker size within colonies. This size diversity increases the division of labour among workers and improves colony fitness. However, in species where size diversity is moderate, this relationship may not always be verified. Here, we manipulated both worker size diversity and mean worker size within colonies of the ant Temnothorax nylanderi, we reintroduced them in the field, and we extensively measured colony fitness after four months. We found no impact on colony survival, growth and reproduction. We discuss how social life and its associated cooperation and conflicts could cause variation in worker size, without any positive effect on colony fitness.