Recently, ensemble-based machine learning models have been widely used and have demonstrated their efficiency in bankruptcy prediction. However, these algorithms are black box models and people cannot understand why they make their forecasts. This explains why interpretability methods in machine learning attract attention from many artificial intelligence researchers. In this paper, we evaluate the prediction performance of Random Forest, LightGBM, XGBoost, and NGBoost (Natural Gradient Boosting for probabilistic prediction) for French firms from different industries with the horizon of 1-5 years. We then use Shapley Additive Explanations (SHAP), a model-agnostic method to explain XGBoost, one of the best models for our data. SHAP can show how each feature impacts the output from XGBoost. Furthermore, single prediction can also be explained, thus allowing black box models to be used in credit risk management.