Women with low levels of vitamin D have a higher risk of developing breast cancer. Numerous studies associated the presence of a CD8+ T cell infiltration with a good prognosis. As vitamin D may play a key role in the modulation of the immune system, the objective of this work was to evaluate the impact of vitamin D on the breast cancer progression and mammary tumor microenvironment. We show that vitamin D decreases breast cancer tumor growth. Immunomonitoring of the different immune subsets in dissociated tumors revealed an increase in tumor infiltrating CD8+ T cells in the vitamin D-treated group. Interestingly, these CD8+ T cells exhibited a more active T cell (TEM/CM) phenotype. However, in high-fat diet conditions, we observed an opposite effect of vitamin D on breast cancer tumor growth, associated with a reduction of CD8+ T cell infiltration. Our data show that vitamin D is able to modulate breast cancer tumor growth and inflammation in the tumor microenvironment in vivo. Unexpectedly, this effect is reversed in high-fat diet conditions, revealing the importance of diet on tumor growth. We believe that supplementation with vitamin D can in certain conditions represent a new adjuvant in the treatment of breast cancers.