Optically switchable field-effect transistors (OSFETs) are non-volatile photonic memory devices holding a great potential for applications in optical information storage and telecommunications. Solution processing of blends of photochromic molecules and π-conjugated polymers is a low-cost protocol to integrate simultaneously optical switching and charge transport functions in large-area devices. However, the limited reversibility of the isomerization of photochromic molecules due to steric hindrance when embedded in ordered polymeric matrices represents a severe limitation and it obliges to incorporate as much as 20% in weight of the photochromic component, thereby drastically diluting the electronic function, limiting the device performance. Herein, a comparative study of the photoresponsivity of a suitably designed diarylethene molecule is reported when embedded in the matrix of six different polymer semiconductors displaying diverse charge transport properties. In particular, this study focuses on three semi-crystalline polymers and three quasi-1D polymers. It is found that 1% w/w of 1,2-bis(5-(3,5-di-tert-butylphenyl)-2-methylthiophen-3-yl)cyclopent-1-ene in a blend with poly(indacenodithiophene-co-benzothiadiazole) is sufficient to fabricate OSFETs combining photo-modulation efficiencies of 45.5%, mobilities >1 cm2 V−1s−1, and photo-recovered efficiencies of 98.1%. These findings demonstrate that quasi-1D polymer semiconductors, because of their charge transport dominated by intra-molecular processes, epitomize the molecular design principles required for the fabrication of high-performance OSFETs.