The latest generation of aerial- and satellite-based imaging sensors acquires huge volumes of Earth’s images with high spatial, spectral and temporal resolution, which open the door to a large range of important applications, such as the monitoring of natural disasters, the planning of urban environments and precision agriculture. In order to fully exploit the potential offered by these sensors, there is a need to develop accurate and time-efficient mathematical models and algorithms for spectral-spatial analysis of the recorded high- resolution data. The main goal of my research is to develop learning approaches, which would help to automatically interpret, or classify, remote sensing images. This manuscript presents several strategies I have explored for this purpose, varying from the use of strong shape priors to detect objects, regularization of classification probabilities on the image graphs, and up to the use of convolutional neural network models capable to learn deep hierar- chical contextual features. The experimental results on diverse benchmarks of images and image time series show the competitiveness of the developed methods when compared to the state-of-the-art ap- proaches. In particular, we have recently created large-scale classification benchmark of aerial images and have demonstrated that the modern deep learning-based methods succeed in generalizing to the dissimilar urban settlements around the Earth. This opens new exciting perspectives towards designing systems which would be able to automati- cally update world-scale maps from remote sensing data.