In this paper, a hybrid features based support vector machine (SVM) model is proposed using infrared thermography technique for hotspots detection and classification of photovoltaic (PV) panels. A novel hybrid feature vector consisting of RGB, texture, the histogram of oriented gradient (HOG), and local binary pattern (LBP) as features is formed using a data fusion approach. A machine learning algorithm SVM is employed to classify the obtained thermal images of PV panels into three different classes (i.e., healthy, non-faulty hotspot, and faulty). The comparison of different machine learning algorithms and datasets is also carried out to validate the superiority of the proposed model and hybrid feature dataset. The experimental results reveal that the proposed hybrid features with SVM resulted in 96.8% training accuracy and 92% testing accuracy with lesser computational complexity and storage space than other machine learning algorithms. The proposed approach is easily implementable for efficient monitoring and fault diagnosis of PV panels.