Fast time domain physical optics (TD-PO) scheme is proposed for calculating the far field of antennas/scatterers exhibiting passive frequency intermodulations. The TD-PO scheme is based on a domain decomposition of the antenna/scaterer surface into smaller patches. The scheme exploits the fact that, the number of temporal and angular samples for accurate description of the far fields is proportional to the size of the patch. Therefore, the radiation patterns are computed directly only for the smallest subdomains. Subsequently, the radiation patterns are aggregated through a multi-level process comprising temporal and angular interpolations and time-shifts, thus, leading to a reduced computational complexity