In this paper, we present a combined approach for human localization and classification in Autonomous Train application. Our contribution is threefold. (a) The creation of a new dataset for workers wearing orange vests in a railway environment context. (b) A deep learning supervised YOLO object detector for persons detection combined with a linear SVM (Support Vector Machine) classifier for persons classification into workers wearing orange vests or travelers. (c) A realtime vision-based technique for the environment monitoring in a driverless train application. Experimental results evaluate the parameters of our two stages detection approach and show that our algorithm is robust in detecting and classifying railway workers for a real-time implementation on an embedded system. Our implementation on an embedded system allows a detection with a correct classification rate of 98.5 % of accuracy and a classification time of 1 ms per frame.