The coupling between unsteady heat release and acoustic perturbations can lead to self-sustained thermoacoustic oscillations, also known as combustion instability. When such combustion instability occurs, the pressure oscillations may become so intense that they can cause engine structural damage and costly mission failure. Thus there is a need to develop a real-time monitoring and control approach, which enables engine systems to be operated stably. In this work, an online monitoring and optimization algorithm is developed to stabilize unstable thermoacoustic systems, which are characterized by nonlinear limit cycle oscillations. It is based on least mean square method (LMS), The performance of the optimization algorithm is evaluated first on a Van der Pol oscillator. It can produce nonlinear limit cycle oscillations, which is similar to pressure oscillation as frequently observed in gas turbine engines. It is shown that implementing the control strategy leads to the oscillations quickly decayed. To further validate the control strategy, experimental study is conducted on a Rijke tube. It is found that approximately 45 dB sound pressure reduction is achieved by actuating a loudspeaker. In addition, the control approach is demonstrated to be able to track and prevent the onset of new limit cycle thermoacoustic oscillations resulting from the changes of fuel flow rate. The present work opens up a new applicable approach to stabilize engine system in terms of minimizing thermoacoustic oscillations. (C) 2016 Elsevier Ltd. All rights reserved.