Rotating detonation combustor (RDC) operation is investigated under the main composition of the coal and supercritical water gasification process (a methane-hydrogen-carbon dioxide mixture), and air is selected as an oxidizer. Experiments are performed under four flow rates (79.1, 101.2, 125.8, and 158.2 g/s), and the equivalence ratio (ER) ranges from 0.23 to 0.87. The detonation delay time and propagation characteristics of the rotating detonation wave (RDW) are investigated, and the nonlinear time series analysis method is applied to the pressure signal using the wavelet entropy algorithm and phase space reconstruction. It is demonstrated that a decrease in the flow rate can increase the detonation delay time near the lean fuel boundary; however, the detonation delay time is affected only slightly when the ER is greater than 0.4. The ER significantly affects the wave velocity but is affected only slightly by the flow rate. Contrary to the wave velocity performance, the pressure of the RDW increases with the flow rate. The lean fuel boundary becomes wider when the flow rate increases. Five combustion modes are investigated: failure, fast deflagration, sawtooth wave, single-wave, and two-counter rotating wave modes. The fast deflagration and sawtooth wave modes may occur near the detonation operating boundary, and a higher flow rate tends to increase the occurrence probability of a multiwave mode. The phase space pattern and distribution of wavelet entropy can be used to distinguish the combustion modes intuitively. (C) 2020 Hydrogen Energy Publications LLC. Published by Elsevier Ltd. All rights reserved.