Ternary organic solar cells contain a single three-component photoactive layer with a wide absorption window, achieved without the need for multiple stacking. However, adding a third component into a well-known binary blend can influence the energetics, optical window, charge carrier transport, crystalline order and conversion efficiency. In the form of binary blends, the low-bandgap regioregular polymer donor poly(3-hexylthiophene-2,5-diyl), known as P3HT, is combined with the acceptor PC61BM, an inexpensive fullerene derivative. Two different non-fullerene acceptors (ITIC and eh-IDTBR) are added to this binary blend to form ternary blends. A systematic comparison between binary and ternary systems was carried out as a function of the thermal annealing temperature of organic layers (100 °C and 140 °C). The power conversion efficiency (PCE) is improved due to increased fill factor (FF) and open-circuit voltage (Voc) for thermal-annealed ternary blends at 140 °C. The transport properties of electrons and holes were investigated in binary and ternary blends following a Space-Charge-Limited Current (SCLC) protocol. A favorable balanced hole–electron mobility is obtained through the incorporation of either ITIC or eh-IDTBR. The charge transport behavior is correlated with the bulk heterojunction (BHJ) morphology deduced from atomic force microscopy (AFM), contact water angle (CWA) measurement and 2D grazing-incidence X-ray diffractometry (2D-GIXRD).