We have studied the effects of Al0.1Ga0.9N(150nm)/AlN composite nucleation layers (CNLs) having different thicknesses of AIN ranging from 20 to 41 nm on the growth characteristics of GaN/Si(1 1 1) epitaxy. The surface morphology of the GaN epitaxial layers which were grown on Al0.1Ga0.9N(150 nm)/AlN CNLs showed that the number of thermal etch pits and cracks were abruptly decreased with the increase of AIN thickness from 20 to 35 run. However, the morphology of GaN epitaxy which was grown on A1(0.1)Ga(0.9)N(150 nm)/AlN CNL having AlN of thickness 41 mn above 35nm showed that the number of etch pits increased again. The GaN/Si(111) epitaxy grown by using A1(0.1)Ga(0.9)N(150nm)/AlN(35nm) CNL showed that the highest crystallinity having a FWHM of 1157arcsec for the (0002) diffraction. Photoluminescence (PL) spectrum at room temperature for GaN/Si(1 1 1) epitaxy grown using Al0.1Ga0.9N(150nm)/AlN(35nm) CNL showed sharp band edge emission at 364nm, which do not have yellow luminescence related to various defects such as vacancy and dislocation. The PL spectra at room temperature for the GaN layers grown using other CNLs showed yellow luminescence at around 580nm in addition to the band edge emission. Moreover, the FWHM of the main exitonic peak at 10K for the GaN/Si(1 1 1) epitaxy, which was grown using Al0.1Ga0.9N(150nm)/AlN(35nm) CNL, has the lowest value of 12.81meV. It is obvious that the Al0.1Ga0.9N(150nm)/AlN CNL having suitable thickness of AlN plays an important role in improving the crystallinity and optical properties of GaN/Si(1 1 1) heteroepitaxy without showing any defects such as pits and cracks over the surface by reducing the mismatch of thermal expansion coefficient and lattice constant between GaN and Si(1 1 1) compared with other nucleation layer such as AlxGa1-xN or AlN alone. (C) 2002 Elsevier Science B.V. All rights reserved.