Two sets of AlInGaN quaternary epilayers grown by metal organic chemical vapor deposition (MOCVD) were studied by atomic force microscopy (AFM), high-resolution X-ray diffraction (HR-XRD) and photoluminescence (PL) spectroscopy. In all samples a large number of pits have been observed on the surface with the density of 1 x 10(9) cm(-2), and hexagonal opening. In one set with different thickness, it was also found that the XRD angle separation between GaN and AlInGaN diffraction peaks increased with the thickness. In PL measurements, a blue shift of the PL peak from AlInGaN was found as the thickness increased. The origin of these observations was proposed to be the Al composition pulling effect. In the other set with different Al/In ratios, domain features were observed on the surface of AlInGaN epilayers with high Al/In ratio. The surface roughness increased as Al/In ratio decreased. PL emission from AlInGaN became stronger as the Al/In ratio approached the lattice-match condition. When the Al/In ratio was below that of lattice-match condition, the emission peak was enhanced and broadened dramatically at the same time, indicating the effect of indium clustering in AlInGaN epilayers. (C) 2004 Elsevier B.V. All rights reserved.