Decaying or annihilating dark matter particles could be detected through gamma-ray emission from the species they decay or annihilate into. This is usually done by modelling the flux from specific dark matter-rich objects such as the Milky Way halo, Local Group dwarfs and nearby groups. However, these objects are expected to have significant emission from baryonic processes as well, and the analyses discard gamma-ray data over most of the sky. Here we construct full-sky templates for gamma-ray flux from the large-scale structure within $\sim$200 Mpc by means of a suite of constrained $N$-body simulations (CSiBORG) produced using the Bayesian Origin Reconstruction from Galaxies algorithm. Marginalising over uncertainties in this reconstruction, small-scale structure and parameters describing astrophysical contributions to the observed gamma ray sky, we compare to observations from the Fermi Large Area Telescope to constrain dark matter annihilation cross-sections and decay rates through a Markov Chain Monte Carlo analysis. We rule out the thermal relic cross-section for $s$-wave annihilation for all $m_\chi \lesssim 7 {\rm \, GeV}/c^2$ at 95% confidence if the annihilation produces $Z$ bosons, gluons or quarks less massive than the bottom quark. We infer a contribution to the gamma ray sky with the same spatial distribution as dark matter decay at $3.3\sigma$. Although this could be due to dark matter decay via these channels with a decay rate $\Gamma \approx 3 \times 10^{-28} {\rm \, s^{-1}}$, we find that a power-law spectrum of index $p=-2.75^{+0.71}_{-0.46}$, likely of baryonic origin, is preferred by the data.