Coronary CT angiography (CCTA) is the only non-invasive imaging technique that reliably depicts the anatomic extent of Coronary Artery Disease (CAD). While occlusion remains a highly predictive indicator of major cardiovascular events (MACE), there is growing evidence that the presence and characteristics of coronary atherosclerosis provide additional prognostic information. In CCTA calcified plaques display high-intensity Hounsfield Units (HU) representative features while more complex representations characterize high-risk soft plaques. As such, accurate identification and quantification is burdensome and time consuming because of the limited temporal, spatial and contrast resolutions of X-ray scanners. Despite the success of deep learning in medical imaging, automatic localization of coronary plaques and especially soft plaques remains a challenging subject in CCTA vessel analysis. For this study, 150 CCTA scans were retrospectively collected. All patients were accepted at triage with minimal to severe CAD suspicion. Selection was carried out with uniform CAD-RADS severity distribution which normally follows an exponential decay function, thus obtaining a higher than normal concentration of plaques. The proposed method outperforms the state of the art for the localization of diverse types of plaques by exploiting the self-attention mechanism of transformers networks to embed contextual features of the coronary tree.