The observed galaxy distribution via galaxy redshift surveys appears distorted due to redshift-space distortions (RSD). While one dominant contribution to RSD comes from the Doppler effect induced by the peculiar velocity of galaxies, the relativistic effects, including the gravitational redshift effect, are recently recognized to give small but important contributions. Such contributions lead to an asymmetric galaxy clustering along the line of sight, and produce non-vanishing odd multipoles when cross-correlating between different biased objects. However, non-zero odd multipoles are also generated by the Doppler effect beyond the distant-observer approximation, known as the wide-angle effect, and at quasi-linear scales, the interplay between wide-angle and relativistic effects becomes significant. In this paper, based on the formalism developed by Taruya et al., we present a quasi-linear model of the cross-correlation function taking a proper account of both the wide-angle and gravitational redshift effects, as one of the major relativistic effects. Our quasi-linear predictions of the dipole agree well with simulations even at the scales below |$20\, h^{-1}\,$|Mpc, where non-perturbative contributions from the halo potential play an important role, flipping the sign of the dipole amplitude. When increasing the bias difference and redshift, the scale where the sign flip happens is shifted to a larger scale. We derive a simple approximate formula to quantitatively account for the behaviours of the sign flip.