We study the influence of the polarization states of femtosecond two-color pulses ionizing gases on the emitted terahertz radiation. A local-current model and plane-wave evaluations justify the previously-reported impact on the THz energy yield and an (almost) linearly-polarized THz field when using circularly-polarized laser harmonics. For such pump pulses, the THz yield is independent on the relative phase between the two colors. When the pump pulses have same helicity, the increase in the THz yield is associated to longer ionization sequences and higher electron transverse momenta acquired in the driving field. Reversely, for two color pulses with opposite helicity, the dramatic loss of THz power comes from destructive interferences driven by the highly symmetric response of the photocurrents lined up on the third harmonic of the fundamental pulse. While our experiments confirm an increased THz yield for circularly polarized pumps of same helicity, surprisingly, the emitted THz radiation is not linearly-polarized. This effect is explained by means of comprehensive 3D numerical simulations highlighting the role of the spatial alignment and non-collinear propagation of the two colors.