The combustion state of rotary kilns during normal burning conditions is critical for entire industrial production processes. However, predicting this state is also challenging. To date, no-one has tried directly using flame images to enable such a prediction. In this paper, a novel neural network architecture is presented that can use a series of flame image sequences with a dynamic spatiotemporal relationship to predict a rotary kiln's combustion state. The proposed neural network architecture implements an end-to-end model of the output that directly draws upon input data, thereby eliminating the need for traditional complicated feature extraction procedures. This method combines the advantages of convolutional neural networks (CNN) and recurrent neural networks (RNN) to facilitate an effective prediction. The method was tested by conducting numerous experiments based on real datasets from a steel plant. The proposed convolutional recurrent neural network (CRNN) achieved an average prediction accuracy of 93.26%, thus verifying that the method is effective and suggesting that it may have the potential for industrial application. (C) 2019 Elsevier Ltd. All rights reserved.