Urban growth is an ongoing trend and one of its direct consequences is the development of buried utility networks. Locating these networks is becoming a challenging task. While the labeling of large objects in aerial images is extensively studied in Geosciences, the localization of small objects (smaller than a building) is in counter part less studied and very challenging due to the variance of object colors, cluttered neighborhood, non-uniform background, shadows and aspect ratios. In this paper, we put forward a method for the automatic detection and localization of manhole covers in Very High Resolution (VHR) aerial and remotely sensed images using a Convolutional Neural Network (CNN). Compared to other detection/localization methods for small objects, the proposed approach is more comprehensive as the entire image is processed without prior segmentation. The first experiments using the Prades-Le-Lez and Gigean datasets show that our method is indeed effective as more than 49% of the ground truth database is detected with a precision of 75%. New improvement possibilities are being explored such as using information on the shape of the detected objects and increasing the types of objects to be detected, thus enabling the extraction of more object specific features.