Self-doped nanocrystals raise great interest for infrared (IR) optoelectronics because their optical properties span from near to far IR. However, their integration for photodetection requires a fine understanding of the origin of their doping and also a way to control the magnitude of the doping. In this paper, we demonstrate that a fine control of the doping level between 0.1 and 2 electrons per dot is obtained through ligand exchange. The latter affects not only the interparticle coupling but also their optical properties because of the band-shift resulting from the presence of surface dipoles. We demonstrate that self-doping is a bulk process and that surface dipoles can control its magnitude. We additionally propose a model to quantify the dipole involved with each ligand. We eventually use the ligand design rule previously evidenced to build a near-infrared photodetector on a soft and transparent substrate. The latter significantly improves the performance compared to previously reported colloidal quantum dot-based photodetectors on plastic substrates operated in the telecom wavelength range.