Context. The Baade-Wesselink (BW) method of distance determination of Cepheids is used to calibrate the distance scale. Various versions of this method are mainly based on interferometry and/or the surface-brightness color relation (SBCR). Aims. We quantify the impact of the SBCR, its slope, and its zero point on the projection factor. This quantity is used to convert the pulsation velocity into the radial velocity in the BW method. We also study the impact of extinction and of a potential circumstellar environment on the projection factor. Methods. We analyzed HARPS-N spectra of η Aql to derive its radial velocity curve using different methods. We then applied the inverse BW method using various SBCRs in the literature in order to derive the BW projection factor. Results. We find that the choice of the SBCR is critical: a scatter of about 8% is found in the projection factor for different SBCRs in the literature. The uncertainty on the coefficients of the SBCR affects the statistical precision of the projection factor only little (1–2%). Confirming previous studies, we find that the method with which the radial velocity curve is derived is also critical, with a potential difference on the projection factor of 9%. An increase of 0.1 in E ( B − V ) translates into a decrease in the projection factor of 3%. A 0.1 mag effect of a circumstellar envelope (CSE) in the visible domain is rather small on the projection factor, about 1.5%. However, we find that a 0.1 mag infrared excess in the K band due to a CSE can increase the projection factor by about 6%. Conclusions. The impact of the surface-brightness color relation on the BW projection factor is found to be critical. Efforts should be devoted in the future to improve the SBCR of Cepheids empirically, but also theoretically, taking their CSE into account as well.