Stability analysis of a Hydro-turbine, an important prime mover in power systems, is meaningful for the safety of both the power station and the power system. However, the structure of hydro-turbines, such as Francis turbine or Kaplan turbine with on-cam mechanism, is complicated which makes it difficult to make an overall estimate of its performance. Considering the hydro-turbine nonlinearity, a comprehensive stability analysis of the hydro-turbine regulation system (HTRS) under frequency control mode (FCM) and power control mode (PCM) is performed. The linear and nonlinear models of the hydroturbine are first established based on Tylor expansion and BP neural network. Then the transfer coefficients in the linear model under full operating conditions (FOC) are computed with the proposed neural-network-based differentiation method (NND). Finally, state space equations of the HTRS with different transfer coefficients are analyzed based on critical stability criterion to explore the variation laws of stability domain and dynamic behaviors of HTRS with operating conditions at FCM and PCM. The results contribute to a better understanding of the influence of hydro-turbine nonlinearity on the stability of HTRS and can be helpful for practical operation in hydropower stations. (C) 2020 Elsevier Ltd. All rights reserved.