This paper is the first in a series where we attempt to define defects in critical lattice models that give rise to conformal field theory topological defects in the continuum limit. We focus mostly on models based on the Temperley-Lieb algebra, with future applications to restricted solid-on-solid (also called anyonic chains) models, as well as non-unitary models like percolation or self-avoiding walks. Our approach is essentially algebraic and focusses on the defects from two points of view: the "crossed channel" where the defect is seen as an operator acting on the Hilbert space of the models, and the "direct channel" where it corresponds to a modification of the basic Hamiltonian with some sort of impurity. Algebraic characterizations and constructions are proposed in both points of view. In the crossed channel, this leads us to new results about the center of the affine Temperley-Lieb algebra; in particular we find there a special basis with non-negative integer structure constants that are interpreted as fusion rules of defects. In the direct channel, meanwhile, this leads to the introduction of fusion products and fusion quotients, with interesting algebraic properties that allow to describe representations content of the lattice model with a defect, and to describe its spectrum.