Structural models such as ontologies and graphs can encode generic knowledge about a scene observed in an image. Their use in spatial reasoning schemes allows driving segmentation and recognition of objects and structures in images. The developed methods include finding a best segmentation path in a graph, global solving of a constraint satisfaction problem, integrating prior knowledge in deformable models, and exploring images in a progressive fashion. Conversely, these models can be specified based on individual information resulting from the segmentation and recognition process. In particular models relying on spatial relations between structures are relevant and more flexible than shape models to be adapted to potential variations, multiple occurrences, or pathological cases. The problem of semantic gap is addressed by generating spatial representations (in the image space) of relations initially expressed in linguistic or symbolic form, within a fuzzy sets formalism. This allows coping with uncertainty and fuzziness, which are inherent both to generic knowledge and to image information. Applications in medical imaging and remote sensing imaging illustrate the proposed paradigm.