Restoring the heterogeneity of agricultural landscapes has been proposed as a key measure to promote farmland biodiversity. Recent studies, however, warn against generalizing this measure because effects can vary across agricultural contexts and species. We explored the hypothesis that heterogeneity has a negative effect on habitat specialists and a positive effect on generalists. We studied 22 species of common farmland birds belonging to three groups: arable specialists, grassland specialists, and mixed habitat (arable, grassland) “generalist” species. Abundances of these three groups were compared on a nationwide gradient of heterogeneity in France (2006 data), including 510 survey sites. We addressed the heterogeneity of the two main farmland habitats: arable land and grassland. We measured habitat extent (arable/(arable + grassland) ratio) and two heterogeneity components: composition (evenness in the land use proportions) and configuration (probability of adjacency). Although maximal configurational heterogeneity was found at maximal compositional heterogeneity, several landscapes had high compositional heterogeneity but low configurational heterogeneity. The abundance of specialists was strongly correlated with habitat extent and negatively correlated with configurational heterogeneity. It suggests that the most important mechanism influencing their population level could be habitat loss, worsened by fragmentation. Generalist species were more abundant in landscapes with higher proportion of arable land and high configurational heterogeneity, which suggests resource supplementation could be the mechanism that improves their population level. Depending on targeted species, opposite effects of heterogeneity can occur. No unique conservation policy solution to maintain all groups of farmland birds exists, promoting landscapes with various heterogeneity levels will be necessary.