Recently, Anomalous Sound Detection (ASD) has emerged as a promising method for road surveillance. However, since the ratio of anomalous events is generally too small, anomaly detection in general, and ASD in particular, are mainly treated as one-class classification problems. Besides, a common problem in road surveillance is the background noise, which makes it difficult to train effective models based on normal sounds only. Therefore, this work aims to experiment with the use of graph signal processing (GSP) to improve ASD performance. Thus, we propose a Graph-based One-Class SVM technique (GOC-SVM) where the features extracted from audio signals are firstly embedded on graphs, and then filtered through a graph filterbank, before computing their joint Fourier transform magnitude. Subsequently, they are fed into a one-class SVM classifier trained on normal data only. Evaluation results show a threefold advantage of using graph embedding and filtering for ASD: (a) improving the anomaly detection results in comparison to plain features, (b) outperforming the classical OC-SVM baseline, (c) enhancing the performance of the proposed semi-supervised GOC-SVM, so as to reach a comparable level of performance of the fully-supervised binary classification SVM, yielding 0.91 of Area-under-the-curve (AUC), 98% of overall accuracy, 99% and 88% of F1 score for normal and anomalous classes, respectively. Such a performance proves the potential of using GSP to solve the ASD problem in road traffic monitoring.