Male homosexual orientation remains a Darwinian paradox, as there is no consensus on its evolutionary (ultimate) determinants. One intriguing feature of homosexual men is their higher male birth rank compared to heterosexual men. This can be explained by two non-exclusive mechanisms: an antagonistic effect (AE), implying that more fertile women have a higher chance of having a homosexual son and to produce children with a higher mean birth rank, or a fraternal birth effect (FBOE), where each additional older brother increases the chances for a male embryo to develop a homosexual orientation due to an immunoreactivity process. However, there is no consensus on whether both FBOE and AE are present in human populations, or if only one of these mechanisms is at play with its effect mimicking the signature of the other mechanism. An additional sororal birth order effect (SBOE) has also recently been proposed. To clarify this situation, we developed theoretical and statistical tools to study FBOE and AE independently or in combination, taking into account all known sampling biases. These tools were applied on new individual data, and on various available published data (two individual datasets, and all relevant aggregated data). Support for FBOE was apparent in aggregated data, with the FBOE increasing linearly with fertility. The FBOE was also supported in two individual datasets. An SBOE is generated when sampling in presence of FBOE, suggesting that controlling for FBOE is required to avoid artefactual SBOE. AE was not supported in individual datasets, including the analysis of the extended maternal family. The evolutionary implications of these findings are discussed.