Design models are essential for many tasks in software engineering, such as consistency checking, code generation and design-to-code tracing. Unfortunately, many UML class diagrams are stored as images, which limits their use and evolution. It is therefore important to identify the semantic elements of design models from images. Although a number of studies focus on the recognition of a UML class diagram, very few address semantic analysis, which is a relatively complex task. In this paper, we propose a framework for training a learning model to categorise and locate semantic elements in class diagram from an image. A large set of annotated design models is proposed and made available online. Qualitative and quantitative evaluations have been carried out on two subsets of data, giving accuracy scores of 92.59% and 94.11% respectively. Evaluations highlight the ability of the proposed learning model to generalise to a wide range of examples.