Solar power stands as a pivotal renewable energy source for the twenty-first century. However, the optimal functioning of solar panels is often hindered by various faults, necessitating accurate and early defect detection to maximize energy production. Existing solar panel fault identification models encounter challenges such as low precision, difficulty in distinguishing fault types, and poor generalization due to limited and unbalanced data samples. This paper introduces a novel and effective approach, leveraging a Binary Cascaded Convolutional Classifier augmented with visual and thermal image combinations to address these limitations. The proposed model adeptly classifies five distinct types of solar panel faults, including single cell hotspots, diode hotspots, dust/ shadow hotspots, multicell hotspots, and Potential-Induced Degradation (PID) hotspots. Through image augmentation techniques like rotation, shifting, sheering, resizing, jittering, and blurring applied to visual and thermal images, inter-class feature variance is increased. Binary Cascaded Convolutional Neural Network (BCCNN) classifiers are trained using an enriched dataset, each specifically designed to differentiate between dust/ shadow hotspots and other fault categories. The adoption of a binary method significantly enhances precision, allowing for focused fault identification and classification. The proposed model surpasses existing literature in terms of precision (99.8%), accuracy (98.5%) and recall (98.4%), underscoring its effectiveness across all five fault classes. In summary, this research marks a substantial advancement in the realm of solar panel fault identification, presenting a more precise and effective fault detection methodology that has the potential to significantly enhance the maintenance and longevity of solar energy systems.