In reactive sputtering, the introduction of reactive gas would create a hysteresis transition from metal to compound mode in both the target and the substrate. The hysteresis transition is characterized by a sudden change in partial pressure, sputtering rate and fraction of compound formation, etc. Therefore, the stability is an important issue of process control. In this paper, a mathematical model with variable sticking coefficients based on surface kinetics is introduced to study the process stability. The variable sticking coefficient represents different mechanisms for surface reactions from the type of Langmuir to precursors. To facilitate the analysis, several nondimensional parameters are introduced and used for formulation. Results show that, when the chemical reaction on a substrate is moderate, a higher sputtering yield of the compound leads to a more stable steady-state at lower inflow rates. However, when the ratio of sputtering yield (compound/metal) is zero, there is no hysteresis transition because of the deposition of pure metal (single phase) on the substrate. For different sticking mechanisms, the precursor type is found to make the surface easier to saturate with compound due to its higher default sticking coefficient and the lower pressure and inflow rate for the hysteresis transition. (C) 2004 Elsevier B.V. All rights reserved.