Nucleation kinetics of ternary nitride layers on GaN substrate has been analysed using classical heterogeneous nucleation theory, incorporating the stress induced supercooling due to lattice mismatch between the substrate and grown layers. Using regular solution model the interfacial tension between the nucleus and substrate and hence interfacial tension between the substrate and mother phase of the compounds have been calculated. The amount of driving force available for the nucleation has been determined for different composition and the degree of supercooling. These values have been used to evaluate the nucleation parameters. It is shown that the nucleation barrier for the formation of ternary nucleus (AlGaN. InGaN and AlInN) on GaN substrate depends strongly on the composition of the alloy.