The automatic prediction of speech intelligibility is a widely known problem in the context of pathological speech. It has been seen as a growing and viable alternative to perceptual evaluation, which is typically time-consuming, highly subjective and strongly biased. Due to this, the development of automatic systems that are able to output not only unbiased predictions, but also interpretable scores become relevant. In this paper we investigate a method to predict speech intelligibility based on consonant phonetic similarity. The proposed methodology relies on a siamese network to compute similarity scores between healthy and pathological phonemes, and based on the combination of those scores, regresses the intelligibility values. Our experimental evaluation suggests a high baseline correlation value of p = 0.82, when applied to our corpus of head and neck cancer. Moreover, further conditioning of the system on specific phonemes in key contexts increased the correlation up to p = 0.89. The given methodology also aims to promote interpretability of the predicted intelligibility score, which is highly relevant in a clinical setting.