Semantic segmentation of remote sensing images is of great significance to the interpretation of remote sensing images. Recently, convolutional neural networks have been increasingly used in this task since it can effectively learn the features in the image. In this paper, an end-to-end semantic segmentation framework, Multi-scale Spatial Aggregation Network (MSAN), is proposed for the remote sensing image segmentation. At first, a classical SegNet is employed as the backbone of the network because its simple structure is suitable for the remote sensing images that have a small quantity of samples. Then several skip connections and a densely connected block are utilized to enhance the usage of the low-level feature and reduce the loss of the detail information in the original image. Moreover, multi-scale spatial information fusion module and a spatial path are added between the encoder and decoder of SegNet, which can effectively extract the features of objects with different sizes in the remote sensing images. Finally, a smoothing algorithm is presented to improve the blocking effect of the remote sensing image segmentation results. The proposed MSAN is tested on the ISPRS Vaihingen dataset and the dataset of a city in southern China, which obtains the satisfactory results.