This paper is a theoretical and a numerical investigation of the stability of a tip-cantilever system used in noncontact atomic force microscopy (NC-AFM) when it oscillates close to a surface. No additional dissipative force is considered. The theoretical approach is based on a variational method exploiting a coarse grained operation that gives the temporal dependence of the nonlinear-coupled equations of motion in amplitude and phase of the oscillator. Stability criterions for the resonance peak are deduced and predict a stable behavior of the oscillator in the vicinity of the resonance. The numerical approach is based on the results obtained with a virtual NC-AFM developed by our group. The effect of the size of the stable domain in phase is investigated. These results are in particularly good agreement with the theoretical predictions. Also they show the influence of the phase shifter in the feedback loop and the way it can affect the damping signal. (C) 2002 Elsevier Science B.V. All rights reserved.