The $Z$-invariant Ising model (Baxter in Philos Trans R Soc Lond A Math Phys Eng Sci 289(1359):315--346, 1978) is defined on an isoradial graph and has coupling constants depending on an elliptic parameter $k$. When $k=0$ the model is critical, and as $k$ varies the whole range of temperatures is covered. In this paper we study the corresponding dimer model on the Fisher graph, thus extending our papers (Boutillier and de Tili{\`e}re in Probab Theory Relat Fields 147:379--413, 2010; Commun Math Phys 301(2):473--516, 2011) to the full $Z$-invariant case. One of our main results is an explicit, local formula for the inverse of the Kasteleyn operator. Its most remarkable feature is that it is an elliptic generalization of Boutillier and de Tili{\`e}re (2011): it involves a local function and the massive discrete exponential function introduced in Boutillier et al. (Invent Math 208(1):109--189, 2017). This shows in particular that $Z$-invariance, and not criticality, is at the heart of obtaining local expressions. We then compute asymptotics and deduce an explicit, local expression for a natural Gibbs measure. We prove a local formula for the Ising model free energy. We also prove that this free energy is equal, up to constants, to that of the $Z$-invariant spanning forests of Boutillier et al. (2017), and deduce that the two models have the same second order phase transition in $k$. Next, we prove a self-duality relation for this model, extending a result of Baxter to all isoradial graphs. In the last part we prove explicit, local expressions for the dimer model on a bipartite graph corresponding to the XOR version of this $Z$-invariant Ising model.