Considered as a weak point in road and railway infrastructure, level crossings (LC) improvement safety became an important field of academic research and took increasingly railways undertakings concerns. Improving safety of people and road-rail facilities is an essential key element to ensure a good operating of the road and railway transport. For this purpose, road and railway safety professionals from several countries have been focused on providing level crossings as safer as possible. Many actions are planned in order to exchange information and provide experiments for improving the management of level crossing safety and performance. This paper aims to develop a video surveillance system to detect, recognize and evaluate potentially dangerous situations in level crossing environments. First, a set of moving objects are detected and separated using an automatic clustering process coupled to an energy vector comparison strategy. Then, a multi-object tracking algorithm, based on optical flow propagation and Kalman filtering correction with adaptive parameters, is implemented. The next step consists on using a Hidden Markov Model to predict trajectories of the detected objects. Finally, the trajectories are analysed with a particular credibility model to evaluate dangerous situations at level crossings. Real data sets are used to test the effectiveness and robustness of the method. This work is developed within the framework of PANsafer project, supported by the French work program ANR.