IEEE Transactions on Automatic Control, Vol.64, No.9, 3632-3646, 2019
State Classification and Multiclass Optimization of Continuous-Time and Continuous-State Markov Processes
We address the long-standing problem of state classification and multiclass optimization of the time-nonhomogeneous continuous-time and continuous-state Markov processes (CTCSMPs). The fundamental property required for state classification is weak ergodicity, with which the state space can be grouped into multiple classes of weak ergodic and branching states. The fundamental property for performance optimization is the state comparability. Optimality conditions are derived for long-run average problems for multiclass CTCSMPs; they take the same form as those for discrete-state Markov processes. It is shown that a stochastic diffusion process is separated by degenerate points into multiple classes. The results also cover the underselectivity issue for long-run average and optimization with nonsmooth value functions. The problem is solved by the relative optimization approach that has been successfully applied to many optimization problems that are not very amenable to dynamic programming.
Keywords:Branching state;direct-comparison-based optimization;relative optimization theory;state comparability;weak ergodic state;weak ergodicity