The field of agriculture research has been transformed by deep learning, which has demonstrated impressive capabilities in detecting and classifying plant disease from leaf images. Although several deep learning-based models are proposed, they are considered as blackbox, lack transparency and are notoriously difficult to interpret. Recently, eXplainable Artificial Intelligence (XAI) methods has demonstrated the potential to interpret the model decision-making process through saliency explanations that highlight the most relevant parts of the input image deemed important for predictions. In this work, we proposed a new XAI saliency method for explanation of potato disease detector based on particular perturbations driven by intermediate object detection results. In order to compare our proposed method with the state of the art, qualitative and quantitative experiments are performed for potato leaf disease detector models on PlantDoc dataset.