Scientific, political and civil society discourses agree that to foster the needed agroecological transition, agricultural diversification is a central element. In this study, we make use of qualitative analysis to examine the visions on agricultural diversification, held by a diversity of actors (farmers, elected officials, civil society, agricultural advisors). The Bauges, a territory located in the French Alps, specialises strongly in extensive dairy cattle breeding to produce Tome des Bauges PDO cheese. This dairy system is, however, increasingly vulnerable to climate change, particularly to droughts. In addition, there is a growing discourse on food autonomy and diversified production. We show that six visions on agricultural diversification, referring to different spatial levels, coexist among the interviewed actors. Two groups of actors can be distinguished based on their vision for, among others, the future of dairy cattle breeding in the Bauges, the possibility of developing other productions, the perimeter for food autonomy and the form of agriculture ecologisation. These two groups tend to oppose conventional breeders, natives of the Bauges, to civil society organisations including neo-resident and diversified farmers in organic farming including neo-rural and “left-returned” farmers. Moreover, the analysis shows that some actors hold plural visions, and that those actors can play different intermediary roles. Thus, to initiate sustainable transformation dynamics at the territorial level, an important step seems to be to highlight these different visions to facilitate dialogue and increase mutual understanding.