The increasing number and resolution of earth observation (EO) imaging sensors has had a significant impact on both the acquired image data volume and the information content in images. There is consequently a strong need for highly efficient search tools for EO image databases and for search methods to automatically identify and recognize structures within EO images. In this paper, we present a concept for an earth observation image data mining system mixing an auto-annotation component with a category search engine which combines a generic image class search and an object detection feature. The proposed concept relies thus on three distinct components which are detailed successively: in the first part, we describe the auto-annotation component, in the second part, the generic category search engine and in the third part, the object detection tool. In the concluding part of the paper, we provide an insight into how these three components can be related to each other and used in a complementary way to arrive at a system which combines the advantages of both the auto-annotation systems and the category search engines.