Breast cancers are commonly associated with an immunosuppressive microenvironment responsible for tumor escape from anti-cancer immunity. Cells of the myeloid lineage account for a major part of this tumor-promoting landscape. These myeloid cells are composed of heterogeneous subsets at different stages of differentiation and have traditionally been described by their cardinal ability to suppress innate and adaptive anticancer immunity. However, evidence has accumulated that, beyond their immunosuppressive properties, breast cancer-induced myeloid cells are also equipped with a broad array of “non-immunological” tumor-promoting functions. They therefore represent major impediments for anticancer therapies, particularly for immune-based interventions. We herein analyze and discuss current literature related to the versatile properties of the different myeloid cell subsets engaged in breast cancer development. We critically assess persisting difficulties and challenges in unequivocally discriminate dedicated subsets, which has so far prevented both the selective targeting of these immunosuppressive cells and their use as potential biomarkers. In this context, we propose the concept of IMCGL, “pro-tumoral immunosuppressive myeloid cells of the granulocytic lineage”, to more accurately reflect the contentious nature and origin of granulocytic cells in the breast tumor microenvironment. Future research prospects related to the role of this myeloid landscape in breast cancer are further considered.