Monitoring urban growth and change is an important task for urban planning and disaster management. While several change detection approaches have been proposed to deal with growing urban areas, their performances are usually limited due to outliers in Satellite Image Time Series (SITS). In this study, in order to discriminate urban growth from the other changes, we exploit spatial connectivity of the changed pixels. To do so, we first stack SITS to a single synthetic image whose pixel values denote the temporal variability along the series. Then, we propose to rely on efficient and well-established spatial filtering by means of the max-tree image representation, leading to a novel approach for detecting changes in urban areas, and more precisely focusing on the spatial extent of such changes in relationship with the urban growth. Experimental results obtained on Landsat imagery of Dar es Salaam showed that our approach helps to remove outliers from the change map and provides satisfactory accuracy.