According to the United Nations, 70% of earth population is going to live in cities by 2050. Given this fast urban evolution, urban monitoring is a key process to qualify sustainable development. Vertical changes need to be assessed, and various methods for 3D change detection have been published. However, there is no common quantitative benchmark assessing their performance in urban areas yet. In this paper, we aim to fill this gap and introduce a simulation tool to generate synthetic 3D point cloud data in a well-controlled scenario. These data are then used to compare qualitatively and quantitatively representative 3D change detection methods for urban areas. These methods are based on distance computation (DSMd, C2C, M3C2), traditional machine learning (RF with stability feature) and deep learning (Feed Forward and Siamese networks). We distinguish between binary and multi-class classification of changes at different levels (3D points, 2D pixels, and 2D patches). While deep neural networks have led to numerous success in remote sensing, we show that they do not systematically outperform more simple methods for 3D change detection. Besides, the existing networks are limited to 2D patches while outputs at the pixel or point scale are more attractive.