Stroke prevention is a major societal issue. Given that 80% of strokes are of ischemic origin, i.e. caused by emboli that can clog brain blood vessels, it is essential to monitor Doppler blood flows in main brain arteries to identify these high intensity transient signals (HITS) and provide a reliable and efficient diagnosis tool to prevent strokes and improve patient care. One area for improvement to help physicians identify the cause of stroke is to assess the nature of the emboli. During cardiac surgery or in neurovascular units dedicated to stroke management, knowing that the embolus is solid or gaseous, or of such a nature, can provide essential clues to the physician to better manage his patients. A strong demand is now identified in this area, but no device offers a reliable solution today. This study proposes the in-vivo HITS classification from time-frequency images using a convolutional network. The major contributions of this study with respect to the previous ones [P. Sombune, IEEE Access, 2018] [A. Tafsast, ICCEE, 2018] are the following: (i) the construction of a database of 1500 HITS issued from 39 patients, (ii) an end-to-end classification of artifacts, solid and gaseous time-frequency in-vivo emboli images using CNN. All Doppler data were acquired with a TCD-X Holter device, a miniaturized, mono-gate and portable device equipped with a robotized probe to maximize alignment between the ultrasound beam and the mean cerebral artery (MCA) throughout the recording period. We studied the performance of HITS classification using the proposed database and different CNN configurations. Additionally, we did dimensionality reduction on the learnt features of the CNN and use the results to improve HITS classification up to 95% of accuracy.