The mathematical modelling in mechanics has a long-standing history as related to geometry, since significant progresses have been achieved by the invention of new geometrical tools. It also sometimes happens that the elucidation of practical issues leads to the invention of new concepts, and possibly new paradigms in science, with impact far beyond. One such example is the intrinsic view of Riemann in geometry with as consequence a radically new insight of physics in the early 20th century. The rather recent intrinsic approaches in elasticity and elasto-plasticity also share this philosophical attitude of looking from inside, or in geometrical language, " in a manifold ". Of course, this vision requires smoothness, and is thus incomplete from an analyst standpoint, but its first aim is to put in the spot light the concepts of metric, curvature and torsion which are addressed in the first part of this paper. In a second step, these concepts are given a precise functional meaning and their properties are studied systematically. Further, a model of elasto-plasticity is designed carrying this intrinsic spirit. The main mathematical object in this approach is the incompatibility operator, a linearized version of Riemann's curvature tensor. So far, this route not only has lead the authors to a new model with a solid functional foundation and proof of existence results, but also to a framework with a minimal amount of ad-hoc assumptions, and complying with both the basic principles of thermodynamics, and invariance principles of physics. The questions arising from this novel approach are complex and intriguing, but we believe that the model is now sufficiently well posed to be studied simultaneously as a problem of mathematics and of mechanics. Most of the research programme remains to be done, and this survey paper is written to present our model, with a particular care to put this approach into a historical perspective.