Taf4 (TATA-box binding protein-associated factor 4) is a subunit of the general transcription factor TFIID, a component of the RNA polymerase II pre-initiation complex that interacts with tissue-specific transcription factors to regulate gene expression. Properly regulated gene expression is particularly important in the intestinal epithelium that is constantly renewed from stem cells. Tissue-specific inactivation of Taf4 in murine intestinal epithelium during embryogenesis compromised gut morphogenesis and the emergence of adult-type stem cells. In adults, Taf4 loss impacted the stem cell compartment and associated Paneth cells in the stem cell niche, epithelial turnover and differentiation of mature cells, thus exacerbating the response to inflammatory challenge. Taf4 inactivation ex vivo in enteroids prevented budding formation and maintenance and caused broad chromatin remodeling and a strong reduction in the numbers of stem and progenitor cells with a concomitant increase in an undifferentiated cell population that displayed high activity of the Ezh2 and Suz12 components of Polycomb Repressive Complex 2 (PRC2). Treatment of Taf4-mutant enteroids with a specific Ezh2 inhibitor restored buddings, cell proliferation and the stem/progenitor compartment. Taf4 loss also led to increased PRC2 activity in cells of adult crypts associated with modification of the immune/inflammatory microenvironment that potentiated Apc-driven tumorigenesis. Our results reveal a novel function of Taf4 in antagonizing PRC2-mediated repression of the stem cell gene expression program to assure normal development, homeostasis, and immune-microenvironment of the intestinal epithelium.