We investigate the growth of mismatched thin films by a kinetic Monte Carlo computer simulation and including a local photoemission model with reflection high-energy electron diffraction (RHEED) intensity for comparison. The strain is introduced through an elastic energy term based on a valence force field approximation. We describe an atomistic mechanism for dislocation nucleation during first stage of GaSb/GaAs (001) growth and in situ variations of photoemission current (PE) and RHEED intensity are reported. We have shown the formation of grooves corresponding to (111) facets, a precursor to the formation of misfit defects. The surface roughening and facetting by creation of grooves explain the absence of photoemission and RHEED oscillations in accordance with experimental observations [J.J. Zinck and D.H. Chow, J. Cryst. Growth, 175/176 (1997) 323, J.J. Zinck and D.H Chow, Appl. Phys. Lett. 66 (1995) 3524]. (C) 2009 Elsevier B.V. All rights reserved.