This article proposes a novel approach to photovoltaic panel inspection through the integration of image classification and meteorological data analysis. Utilizing two convolutional neural network models with distinct architectures for classifying thermal and red, green, blue (RGB) images of photovoltaic installations, in addition to an support vector machines model for meteorological data classification, the results from these models are concatenated, allowing the fusion of visual and meteorological information for comprehensive defect detection. Data collection from photovoltaic panels is achieved using a portable device, followed by the application of advanced image processing techniques to identify faults rapidly and accurately with up to 96% accuracy. The inspection results are presented in a user-friendly format, facilitating straightforward interpretation and analysis. This new approach has the potential to significantly enhance the efficiency and durability of solar energy systems, enabling timely maintenance and repair for photovoltaic panel issues.