In neuroscience, learning and memory are usually associated to long-term changes of connection strength between neurons. In this context, synaptic plasticity refers to the set of mechanisms driving the dynamics of neuronal connections, called synapses and represented by a scalar value, the synaptic weight. A Spike-Timing Dependent Plasticity (STDP) rule is a biologically-based model representing the time evolution of the synaptic weight as a functional of the past spiking activity of adjacent neurons. If numerous models of neuronal cells have been proposed in the mathematical literature, few of them include a variable for the time-varying strength of the connection. In this article, a new, general, mathematical framework to study the phenomenon of synaptic plasticity associated to STDP rules is introduced. A system composed of two neuronal cells connected by a single synapse is investigated and a stochastic process describing its dynamical behavior is presented and analyzed. The notion of plasticity kernel is introduced as a key component of plastic neural networks models. We show that a large number of STDP rules from neuroscience and physics applied to neural systems can be represented by this formalism. Experiments show that long-term synaptic plasticity evolves on a much slower timescale than the cellular mechanisms driving the activity of neuronal cells. For this reason a scaling model of our stochastic model is also introduced and averaging principles for a sub-class of plasticity kernels are stated, and proved in a companion paper. These results are used to analyze two STDP models widely used in applied physics: Pair-based rules and calcium-based rules. We compare results of computational neuroscience on models of timing-based synaptic plasticity with our results derived from averaging principles.