In this paper, we studied the roughening of SiGeC surface revealed by in situ reflection high energy electron diffraction (RHEED) measurements, that happens during ultra-high vacuum chemical vapor deposition (UHV-CVD) growth under certain growth conditions. A high growth rate and a low temperature are found to be favorable for smooth surfaces. Roughening is accompanied by a dramatic decrease of the substitutional C content and, further, stacking faults develop within the epilayer. According to these observations, we proposed a model of surface roughening based on the formation of carboneous complexes on the film surface and a way to maximize substitutional C incorporation in very thin layers by using RHEED as a probe for monitoring the development of lattice defects. By this means, more than 2% of substitutional C atoms can be incorporated in 5 nm Si1-yCy films (assuming Vegard's law). (C) 2003 Elsevier B.V. All rights reserved.