Cloud cover and cloud motion have a large impact on solar irradiance. One of the effective ways for direct normal irradiance (DNI) prediction is to use cloud features, which has been extensively studied. A Convolutional Neural Network (CNN) has the advantage of automatic features extraction by using strong computing capabilities. In this paper, a novel 3D-CNN method is proposed by processing multiple consecutive ground-based cloud (GBC) images in order to extract cloud features including texture and temporal information. The resulting features and the DNI data are then used to establish a DNI forecasting model. The experiments are carried out to evaluate the performance of the proposed forecasting method by using the data from January 1, 2013 to December 31, 2014. The experimental results show that the proposed method coupled with the multilayer perceptron (MLP) model achieves forecast skill of 17.06% for 10-minute ahead DNI prediction.