Perturbative and non-perturbative expansion methods already constitute a tool of choice to perform ab initio calculations over a significant part of the nuclear chart. In this context, the categories of accessible nuclei directly reflect the class of unperturbed state employed in the formulation of the expansion. The present work generalizes to the nuclear many-body context the versatile method of Burton and Thom (J Chem Theory Comput 16(4):5586, 2020) by formulating a perturbative expansion on top of a multi-reference unperturbed state mixing deformed non-orthogonal Bogoliubov vacua, i.e. a state obtained from the projected generator coordinate method (PGCM). Particular attention is paid to the part of the mixing taking care of the symmetry restoration, showing that it can be exactly contracted throughout the expansion, thus reducing significantly the dimensionality of the linear problem to be solved to extract perturbative corrections. While the novel expansion method, coined as PGCM-PT, reduces to the PGCM at lowest order, it reduces to single-reference perturbation theories in appropriate limits. Based on a PGCM unperturbed state capturing (strong) static correlations in a versatile and efficient fashion, PGCM-PT is indistinctly applicable to doubly closed-shell, singly open-shell and doubly open-shell nuclei. The remaining (weak) dynamical correlations are brought consistently through perturbative corrections. This symmetry-conserving multi-reference perturbation theory is state-specific and applies to both ground and excited PGCM unperturbed states, thus correcting each state belonging to the low-lying spectrum of the system under study. The present paper is the first in a series of three and discusses the PGCM-PT formalism in detail. The second paper displays numerical zeroth-order results, i.e. the outcome of PGCM calculations. Second-order, i.e. PGCM-PT(2), calculations performed in both closed- and open-shell nuclei are the object of the third paper.