This paper proposes a novel deep learning model called deep multi-scale network (DMSN) for background subtraction. This convolutional neural network is built to use RGB color channels and Depth maps as inputs with which it can fuse semantic and spatial information. In comparison with previous deep learning background subtraction techniques that lack information due to its use of only RGB channels, our RGBD version is able to overcome most of the drawbacks, especially in some particular kinds of challenges. Further, this paper introduces a new protocol for the SBM-RGBD dataset, concerning scene-independent evaluation, dedicated to Deep Learning methods to set up a competitive platform that includes more challenging situations. The proposed method proved its efficiency in solving the background subtraction in complex situations at different levels. The experimental results verify that the proposed work outperforms the state of the art on SBM-RGBD and GSM datasets.