As an essential issue in wind energy industry, wind speed forecasting plays a vital role in optimal scheduling and control of wind energy generation and conversion. In this paper, a novel method called EnsemLSTM is proposed by using nonlinear-learning ensemble of deep learning time series prediction based on LSTMs (Long Short Term Memory neural networks), SVRM (support vector regression machine) and EO (extremal optimization algorithm). First, in order to avert the drawback of weak generalization capability and robustness of a single deep learning approach when facing diversiform data, a cluster of LSTMs with diverse hidden layers and neurons are employed to explore and exploit the implicit information of wind speed time series. Then predictions of LSTMs are aggregated into a nonlinear-learning regression top-layer composed of SVRM and the EO is introduced to optimize the parameters of the top-layer. Lastly, the final ensemble prediction for wind speed is given by the fine-turning top-layer. The proposed EnsemLSTM is applied on two case studies data collected from a wind farm in Inner Mongolia, China, to perform ten-minute ahead utmost short term wind speed forecasting and one-hour ahead short term wind speed forecasting. Statistical tests of experimental results compared with other popular prediction models demonstrated the proposed EnsemLSTM can achieve a better forecasting performance.