The specificity of dairy Protected Designation of Origin (PDO) products is related to their “terroir” of production. This relationship needs better understanding for efficient and sustainable productions preserving the agroecological equilibrium of agroecosystems, especially grasslands. Specificity of PDO Comté cheese was related to the diversity of natural raw milk bacterial communities, but their sources need to be determined. It is hypothesized that raw milk indigenous microbial communities may originate from permanent grazed grasslands by the intermediate of dairy cows according to the sequence soil–phyllosphere–teat–milk. This hypothesis was evaluated on a 44 dairy farms network across PDO Comté cheese area by characterizing prokaryotic and fungal communities of these compartments by metabarcoding analysis (16S rRNA gene: V3–V4 region, 18S rRNA gene: V7–V8 region). Strong and significant links were highlighted between the four compartments through a network analysis (0.34 < r < 0.58), and were modulated by soil pH, plant diversity and elevation; but also by farming practices: organic fertilization levels, cattle intensity and cow-teat care. This causal relationship suggests that microbial diversity of agroecosystems is a key player in relating a PDO product to its “terroir”; this under the dependency of farming practices. Altogether, this makes the “terroir” even more local and needs to be considered for production sustainability.