We construct the Baxter operator Q(λ) for the q-Toda chain and the Toda2 chain (the Toda chain in the second Hamiltonian structure). Our construction builds on the relation between the Baxter operator and Bäcklund transformations that were unravelled in [13]. We construct a number of quantum intertwiners ensuring the commutativity of Q(λ) with the transfer matrix of the models and of the Q’s between each other. Most importantly, Q(λ) is modular invariant in the sense of Faddeev. We derive the Baxter equation for the eigenvalues q(λ) of Q(λ) and show that these are entire functions of λ. This last property will ultimately lead to the quantization of the spectrum for the considered Toda chains, in a subsequent publication [1]. This work is dedicated to the memory of L. D. Faddeev