Blind adaptive source separation (BASS) based compensation for transmitter (Tx) IQ imbalance is presented for the first time in an M-QAM optical coherent system. The proposed method is numerically investigated with 4-QAM and 16-QAM signals in the presence of Tx IQ imbalance up to 30°. The robustness of the BASS method is studied after 200-km optical fiber transmission, in which the effects of chromatic dispersion (CD) and carrier frequency offset (CFO) are assumed to be dominant. It is also found that CFO, inherent to frequency difference between the transmitter and receiver lasers in optical coherent transmission, should be compensated before IQ imbalance compensation to achieve a better performance. The proposed method outperforms the Gram-Schmidt orthogonalization procedure (GSOP) in the presence of CD and CFO. We further validate experimentally the proposed method with 10-Gbaud optical 4-QAM and 16-QAM signals at 30° and 10° phase imbalance, respectively, with an emulated 200-km optical fiber transmission and 200-MHz CFO. More specifically, the optical signal-to-noise ratio (OSNR) penalty reduction of the BASS method compared to the GSOP method is 1 dB for 4-QAM at a bit-error-ratio (BER) of 2.10^-3 and 2 dB for 16-QAM at a BER of 10^-3. Moreover, instead of being a fully independent block and requiring statistical estimation as in GSOP, the BASS method can be integrated into an equalizer and operated at the sample rate, simplifying the operation and allowing parallel implementation.