With the construction of cascaded hydropower stations and high-voltage direct current transmission systems, hydro-dominated power systems are being developed all over the world. Regulation stability of sending terminals is an important guarantee for utilizing clean energy. However, the potential hydropower-induced frequency oscillations in these systems might limit the hydropower delivery quality. There is a need for comprehensive understanding and systematic tools to support decision making processes in frequency regulation. This paper studies this issue from three aspects: modelling adaptability, quantitative analysis and coordinated optimization. Firstly, four models of hydropower governor system in two regulation modes are conducted to analyze the effect of model simplification on stability. The refined eighth-order model of hydraulic damping is derived. Secondly, the oscillation characteristics are quantified by applying a new method of stability margin. The operating boundary is proposed to guide the high-quality operation. Finally, a novel multi-objective optimization strategy is proposed to coordinate the stability-tracking conflict. The reliability and effectiveness are demonstrated in the isolated operation mode and the islanded operation mode. This work can be applied by power producers and grid operators to a better understanding of oscillation features, and supply insights and method on how to improve the regulation quality of hydro-dominated system. (c) 2020 Elsevier Ltd. All rights reserved.