The mathematical modelling in mechanics has a long-standing history as related to geometry, and significant progresses have often been achieved by the invention of new geometrical tools. Also, it happened that the elucidation of practical issues led to the invention of new scientific concepts, and possibly new paradigms, with potential impact far beyond. One such example is Riemann's intrinsic view in geometry, that offered a radically new insight in the Physics of the early 20-th century. On the other hand, the rather recent intrinsic approaches in elasticity and elasto-plasticity also share this philosophical standpoint of looking from inside, i.e., from the "manifold" point of view. Of course, this approach requires smoothness, and is thus incomplete for an analyst. Nevertheless, its first aim is to highlight the concepts of metric, curvature and torsion; these notions are addressed in the first part of this survey paper. In a second part, they are given a precise functional meaning and their properties are studied systematically. Further, a novel approach to elasto-plasticity constructed upon a model of incompatible elasticity is designed, carrying this intrinsic spirit. The main mathematical object in this theory is the incompatibility operator, i.e., a linearized version of Riemann's curvature tensor. So far, this route not only has led the authors to a new model with a solid functional foundation and proof of existence results, but also to a framework with a minimal amount of ad-hoc assumptions, and complying with both the basic principles of thermodynamics and invariance principles of Physics. The questions arising from this novel approach are complex and intriguing, but we believe that the model is now sufficiently well-posed to be studied simultaneously as a problem of mathematics and of mechanics. Most of the research program remains to be done, and this survey paper is written to present our model, with a particular care to put this approach into a historical perspective.