The photovoltaic/thermal (PVT) system is developed to combine the two separate systems to reduce the located area and cost, improve the efficiency and save lots of materials. However, different PVT designs and configurations had been proposed with air, water, nanofluid, phase change materials (PCM) and mix of these cooling methods. In this paper, a mathematical model was proposed for the new nanofluid/nano-PCM PVT system. Experiments had been conducted to validate the mathematical model results. The comparison confirmed the ability of the proposed mathematical model to satisfactory simulate and match the experimental results. The experimental and mathematical model results show that electrical and thermal efficiencies are 13.7%, 13.2% and 72%, 71.3% respectively. However, it is found that the maximum temperature registered in glass, PV cell, wax, and nanofluid are 41.2, 39.92, 38.8 and 36.5, respectively. The proposed mathematical model accuracy compared and validated with experiment results and shows a good consistent and agreement.