Fires are among the most frequent disasters, causing serious injuries and extensive property destruction. In order to prevent the uncontrolled spread of fires, recognizing fires accurately and at an early stage is crucial, especially in video surveillance applications. The majority of the available deep fire detection models currently operate on single images, limiting their analysis to spatial features only. The temporal context and motion information, present in consecutive frames of a scene, are not involved leading to incorrect predictions throughout the video. To address this shortcoming, it is proposed in this work to explore the temporal information using deep learning networks to directly recognize fire. Indeed, a novel three-dimensional convolutional neural network, named 3D Fire Classification Network, is introduced. This approach exploits spatio-temporal features to analyze and recognize a video sequence as either fire or non-fire. Initially, the input data is processed to enlarge and diversify the constructed dataset. Then, it is passed through the designed network for training. The derived model com- prises a relatively smaller number of layers, with a reduced number of parameters. The conducted experiments demonstrate the efficiency of the resulting model on the created dataset, achieving an improved accuracy of 99.23%. Furthermore, the findings show that the developed model consistently outperforms the related methods in recognizing fire videos.