Background and objective: 3D rotational angiography (3DRA) provides high quality images of the cerebral arteriovenous malformation (AVM) nidus that can be reconstructed in 3D. However, these reconstructions are limited to only 3D visualization without possible interactive exploration of geometric characteristics of cerebral structures. Refined understanding of the AVM angioarchitecture prior to treatment is mandatory and vascular segmentation is an important preliminary step that allow physicians analyze the complex vascular networks and can help guide microcatheters navigation and embolization of AVM. Methods: A deep learning method was developed for the segmentation of 3DRA images of AVM patients. The method uses a fully convolutional neural network with a U-Net-like architecture and a DenseNet backbone. A compound loss function, combining Cross Entropy and Focal Tversky, is employed for robust segmentation. Binary masks automatically generated from region-growing segmentation have been used to train and validate our model. Results: The developed network was able to achieve the segmentation of the vessels and the malformation and significantly outperformed the region-growing algorithm. Our experiments were performed on 9 AVM patients. The trained network achieved a Dice Similarity Coefficient (DSC) of 80.43%, surpassing other U- Net like architectures and the region-growing algorithm on the manually approved test set by physicians. Conclusions: This work demonstrates the potential of a learning-based segmentation method for characterizing very complex and tiny vascular structures even when the training phase is performed with the results of an automatic or a semi-automatic method. The proposed method can contribute to the planning and guidance of endovascular procedures.