CeO2 layers with various degrees of crystallinity are characterized to get insight into the origin of the deterioration of epitaxial layers on Si substrates. The surface morphology of CeO2 layers is studied using atomic force microscopy and is correlated with the crystallinity of the layer determined by reflection high-energy electron diffraction. It is clearly observed that the surface morphology changes according to the crystallinity of the CeO2 layers. Single crystalline CeO2(110) layers with good crystallinity have a nanometer-scale periodically corrugated surface structure, which consists of (111) facets. As the crystalline quality of the CeO2 layers becomes worse, the number of irregularly shaped hillocks and tetrahedral hillocks increases and corresponds to distorted (110) grains and (111)-oriented polycrystalline grains, respectively. It is found that the crystalline quality is not uniform; different regions of crystallinity are distributed from place to place and the ratio of their population changes according to the degree of crystalline quality. Direct observation of irregularly shaped hillocks is carried out using cross-sectional lattice images from high resolution transmission electron microscopy. These features are discussed to understand the origin of the irregularly shaped hillocks in terms of the surface properties of the substrates at the early stage of growth. (C) 2003 American Vacuum Society.