In this work, we present various machine learning models to predict emotional states in horses. We manually label the images to learn the task in a supervised manner. We perform data exploration and use different cropping methods, mainly based on Yolo and Faster R-CNN, to create two new datasets: 1) the cropped body, and 2) the cropped head dataset. We build different models based on convolutional neural networks (CNNs) using (un)cropped datasets and compare their performances to accurately predict emotions. The cropped head dataset yields the best results despite lacking important region of interests like a tail, which experts use to annotate images. Furthermore, we update our models using various techniques, such as transfer learning and fine-tuning to further improve their performance. The best performance is achieved through a model based on stacking principals, which gave a boost to the overall performance with an accuracy of 87%, precision of 79%, and recall of 97%. Finally, we employ three interpretation methods to understand the internal workings of our models. Different interpretation methods seem to highlight different features of the same model. We found that only LIME appears to detect some of the features that are used by experts to annotate emotional states.