Raster-based map generalization is nowadays anecdotal, as most generalization operations are performed using vector data. Vectors describe the shape of each object in the map using a set of coordinates; thus, the object delimitation is directly accessible, and the topology and distance-based relations are easy to compute. On the contrary, rasters represent a map as an image, a grid of pixel covers the target area, and each pixel is characterised by a value. This representation does not explicitly model the boundary/shape of geographic objects and the relations between them. However, the emergence of the image-based deep learning techniques has shown an ability to process images of geographic information. The question of their adaptation for map generalization is a trendy subject: road (Courtial et al. 2020), building (Feng et al. 2019) and coastline (Du et al. 2021) generalization have been explored in recent years. Common methods for evaluating these techniques seems to be necessary for the comparison and development of this field. Here we propose to create a benchmark for comparing raster-based methods for mountain road generalization