A nanostructured surface layer with a thickness up to 60 mu m was produced on a rare earth Mg-Gd-Y magnesium alloy using a new process named HVOF-SMB (high velocity oxygen-fuel flame supersonic microparticles bombarding). The microstructural features of the treated surface at various depth of the deformed layer were characterized by optical microscopy (OM), transmission electron microscopy (TEM) and high-resolution transmission electron microscopy (HRTEM) with an aim to reveal the formation mechanism. Results showed that three steps during grain refinement process were found, i.e., twinning dominates the plastic deformation and divides the coarse grains into finer twin platelets at the initial stage, stacking faults are generated and a number of dislocation slip systems are activated leading to the cross slips with increasing strain and strain rate, eventually high-density dislocation networks, dislocation cells and dislocation arrays are formed, which further subdivides the twin platelets and residual microbands into sub-microstructures. As a result, homogeneous nanostructure with a grain size of about 10-20 nm is formed through dynamic recrystallization in the topmost surface layer. Based on the experimental observations, a grain refinement mechanism induced by plastic deformation with higher strain rate during the HVOF-SMB treatment in the rare earth Mg-Gd-Y alloy was proposed. (c) 2009 Elsevier B.V. All rights reserved.