In the context of content-based indexing applications, the automatic classification and interpretation of visual content is a key issue that needs to be solved. This paper proposes a novel approach for semantic video object interpretation. The principle consists of exploiting the a priori information contained in categorized 3D model data sets, in order to transfer the semantic labels from such models to unknown video objects. Each 3D model is represented as a set of 2D views, described with the help of shape descriptors. A matching technique is used in order to perform an association between categorized 3D models and 2D video objects. The experimental evaluation shows the interest of our approach, which yields recognition rates of up to 92.5%.