Glioma grading before surgery is very critical for the prognosis prediction and treatment plan making. We present a novel wavelet scattering-based radiomic method to predict noninvasively and accurately the glioma grades. The method consists of wavelet scattering feature extraction, dimensionality reduction, and glioma grade prediction. The dimensionality reduction was achieved using partial least squares (PLS) regression and the glioma grade prediction using support vector machine (SVM), logistic regression (LR) and random forest (RF). The prediction obtained on multimodal magnetic resonance images of 285 patients with well-labeled intratumoral and peritumoral regions showed that the area under the receiver operating characteristic curve (AUC) of glioma grade prediction was increased up to 0.99 when considering both intratumoral and peritumoral features in multimodal images, which represents an increase of about 13% compared to traditional radiomics. In addition, the features extracted from peritumoral regions further increase the accuracy of glioma grading.