Colorectal cancer is a global public health problem with one of the highest death rates. It is the second most deadly type of cancer and the third most frequently diagnosed in the world. The present study focused on metastatic colorectal cancer (mCRC) patients who had been treated with chemotherapy-based regimen for which it remains uncertainty about the efficacy for all eligible patients. This is a major problem, as it is not yet possible to test different therapies in view of the consequences on the health of the patients and the risk of progression. Here, we propose a method to predict the efficacy of an anticancer treatment in an individualized way, using a deep learning model constructed on the retrospective analysis of the primary tumor of several patients. Histological sections from tumors were imaged by standard hematoxylin and eosin (HE) staining and infrared spectroscopy (IR). Images obtained were then processed by a convolutional neural network (CNN) to extract features and correlate them with the subsequent progression-free survival (PFS) of each patient. Separately, HE and IR imaging resulted in a PFS prediction with an error of 6.6 and 6.3 months respectively (28% and 26% of the average PFS). Combining both modalities allowed to decrease the error to 5.0 months (21%). The inflammatory state of the stroma seemed to be one of the main features detected by the CNN. Our pilot study suggests that multimodal imaging analyzed with deep learning methods allow to give an indication of the effectiveness of a treatment when choosing.