Predictions of the mean and covariance matrix of summary statistics are critical for confronting cosmological theories with observations, not least for likelihood approximations and parameter inference. Accurate estimates require running costly N-body and hydrodynamics simulations. Approximate solvers, or surrogates, greatly reduce the computational cost but introduce biases, especially in the non-linear regime of structure growth. We propose ‘CARPool Bayes’ to solve the inference problem for both the means and covariances using a combination of simulations and surrogates. Our approach allows incorporating prior information for the mean and covariance. We derive closed-form solutions for maximum a posteriori covariance estimates that are efficient Bayesian shrinkage estimators, guarantee positive semidefiniteness, and can optionally leverage analytical covariance approximations. We discuss choices of the prior and propose a procedure for obtaining optimal prior hyperparameter values with a small set of test simulations. We test our method by estimating the covariances of clustering statistics of gadget-iiiN-body simulations at redshift z = 0.5 using surrogates from a 100–1000× faster particle-mesh code. Taking the sample covariance from 15 000 simulations as the truth, and using an empirical Bayes prior with diagonal blocks, our estimator produces nearly identical Fisher matrix contours for ΛCDM parameters using only 15 simulations of the non-linear dark matter power spectrum. In this case, the number of simulations is so small that the sample covariance is degenerate. We show cases where even with a naïve prior our method improves the estimate. Our framework is applicable to a wide range of cosmological problems where fast surrogates are available.