Safety and security are the most discussed topics in the road and railway transportation field. Latest security initiatives in the field of railway transportation propose to implement video surveillance at level crossing (LC) environments. In this paper we explore the possibility of implementing a smart video surveillance security system that is tuned toward detecting and evaluating abnormal situations induced by users (pedestrians, vehicle drivers, and unattended objects) in LCs. This intelligent security system starts by detecting, separating, and tracking moving objects shot in the LC. Then, a hidden Markov model is developed to estimate ideal trajectories, allowing the detected targets to discard dangerous situations. After that, the level of risk of each target is instantly estimated by using the Dempster-Shafer data fusion technique. The proposed analysis allows for also recognizing hazard scenarios. The video surveillance system is connected to a communication system (the Wireless Access for Vehicular Environment), which takes the information on the dynamic status of the LC (safe or presence of a dangerous situation) and sends it to users approaching the LC. Four hazard scenarios are tested and evaluated with different real video image sequences: presence of the obstacle in the LC, presence of the stopped vehicles line, vehicle zigzagging between two closed half barriers, and pedestrian crossing the LC area.