Prior to any numerical development, the paper objective is to answer first to a fundamental question: what is the mathematical form of the most general data-driven constitutive model for stable materials, taking maximum account of knowledge from physics and materials science? Here we restrict ourselves to elasto-(visco-)plastic materials under the small displacement assumption. The experimental data consists of full-field measurements from a family of tested mechanical structures. In this framework, a general data-driven approach is proposed to learn the constitutive model (in terms of thermodynamic potentials) from data. A key element that defines the proposed data-driven approach is a tool: the Constitutive Relation Error (CRE); the data-driven model is then the minimizer of the CRE. A notable aspect of this procedure is that it leads to quasi-explicit formulations of the optimal constitutive model. Eventually, a modified Constitutive Relation Error is introduced to take measurement noise into account.