To meet the challenge of handling dynamic systems in which continuous dynamics and discrete events coexist, we develop stability analysis of such systems governed by ordinary differential equations and stochastic differential equations with regime switching modulated by a Markov chain involving a small parameter. The small parameter is used to reflect either the inherent two-time scales in the original system or the different rates of regime switching among a large number of states of the discrete events. The smaller the parameter is, the more rapid switching the system will experience. To reduce the complexity, one attempts to effectively "replace" the actual systems by a limit system with simpler structure. To ensure the validity of such a replacement in a longtime horizon, it is crucial that the original system is stable. The fast regime changes and the large state space of the Markov chain make the stability analysis difficult. Under suitable conditions, using the limit dynamic systems and the perturbed Lyapunov function methods, we show that if the limit systems are stable, then so are the original systems. This justifies the replacement of a complicated original system by its limit from a longtime behavior point of view.