There has been an exponential increase in Photovoltaic energy over the last decade. The size and the complexity of photovoltaic solar power plants are increasing, and it requires advanced and robust condition monitoring systems for ensuring their reliability. To this aim, a novel method is addressed for fault detection in photovoltaic panels through processing of thermal images of solar panels captured by a thermographic camera. In this paper, two advanced convolutional neural network models are used wherein the task of the first model is to classify the type of fault affecting the panel and the task of the second model is to identify the region of interest of the faulty panel. Proposed approach uses F1 score as a metric to compare several classification models of which the ResNet-50 transfer learning model achieves the highest score of 85.37 %. Mean Average Precision is used as an evaluation metric for object detection models wherein the highest scoring model is Faster R-CNN with a score of 67 %. This paper puts forth an approach to facilitate early identification and fault localization in Solar Panels by minimizing the amount of manual labour involved in the process