Due to the population growth and the increased reliance on cooling and heating systems, buildings have become the largest energy consumer worldwide. The use of phase change material (PCM) has shown great potential to reduce the annual cooling and heating load by up to 50%. Nowadays, the direct incorporation of PCM in cement-based materials (CBM) is creating a considerable debate in the research community with regards to the proper selection and the beneficial utilization of PCM (microencapsulated or composite) in CBM. Therefore, this paper reviews the pros and cons of using microencapsulated and composite PCM in CBM by highlighting the mechanisms involved in the mechanical strength loss and thermal properties enhancement. Generally, a high thermal energy storage CBM was obtained. However, PCM exhibited a negative effect on the compressive strength of CBM. In view of the literature review, the compressive strength reduction varies considerably with no clear trend which is understandable in view of the differences in mix designs as well as the variety of materials used in each study. Finally, an up-to-date PCM case studies, gaps and future directions are also presented to provide a reliable basis and helpful reference for the future development of eco-friendly and energy-efficient building materials containing PCM.