In this article, we prove that Ramsey's theorem for pairs and two colors is a $\forall \Pi^0_4$ conservative extension of~$\mathsf{RCA}_0 + \mathsf{B}\Sigma^0_2$, where a $\forall \Pi^0_4$ formula consists of a universal quantifier over sets followed by a $\Pi^0_4$ formula. The proof is an improvement of a result by Patey and Yokoyama and a step towards the resolution of the longstanding question of the first-order part of Ramsey's theorem for pairs.