Due to the black-box nature of many artificial intelligence models, their safety certification using traditional methods, process and standards such as the EN 50128 is often impossible. For this reason, the increasing demand to allow usage of AI systems in railway’s safety related function requires the development of new processes to certificate their safety. One of the proposed approaches toward the certification of AI systems is to allow their usage on the basis of a statistical safety demonstration. In this paper, we discuss the implications and some issues of such a safety demonstration on the basis of discussions with researcher, industrials and representatives of authorizing agencies and using an AI based autonomous train perception system as an example.