The next generation of galaxy surveys has the potential to substantially deepen our understanding of the Universe. This potential hinges on our ability to rigorously address systematic uncertainties. Until now, diagnosing systematic effects prior to inferring cosmological parameters has been out of reach in field-based implicit likelihood cosmological inference frameworks. As a solution, we aim to diagnose a variety of systematic effects in galaxy surveys prior to inferring cosmological parameters, using the inferred initial matter power spectrum. Our approach is built upon a two-step framework. First, we employ the Simulator Expansion for Likelihood-Free Inference (SELFI) algorithm to infer the initial matter power spectrum, which we utilise to thoroughly investigate the impact of systematic effects. This investigation relies on a single set of N-body simulations. Second, we obtain a posterior on cosmological parameters via implicit likelihood inference, recycling the simulations from the first step for data compression. For demonstration, we rely on a model of large-scale spectroscopic galaxy surveys that incorporates fully non-linear gravitational evolution and simulates multiple systematic effects encountered in real surveys. We provide a practical guide on how the SELFI posterior can be used to assess the impact of misspecified galaxy bias parameters, selection functions, survey masks, inaccurate redshifts, and approximate gravity models on the inferred initial matter power spectrum. We show that a subtly misspecified model can lead to a bias exceeding $2\sigma$ in the $(\Omega_\mathrm{m},\sigma_8)$ plane, which we are able to detect and avoid prior to inferring the cosmological parameters. This framework has the potential to significantly enhance the robustness of physical information extraction from full-forward models of large-scale galaxy surveys such as DESI, Euclid, and LSST.