Due to the development of energy performance contracting and the needs for peak electric demand reduction, the interest for building energy demand prediction is renewed. Gray-box models are a solution for energy demand prediction. However, it is still difficult to find the best level of model complexity and the good practices for the training phase. Since models' order and parameter identification method have a strong impact on the forecasting precision and are not intuitive, a comparative design approach is used to find the best model architecture and an adequate methodology for improving the training phase, The gray box models are compared on their ability to forecast heating and cooling demands and indoor air temperature. An objective function is proposed aiming to minimize both power and indoor temperature prediction errors. Moreover, for each model, several training period durations are tested. First, this study shows that a R6C2 (second order model) model is adapted to predict the building thermal behavior. Furthermore, the best fits are obtained with two weeks of data for the identification process. Second, a sensitivity analysis using total Sobol index calculation leads to validate the objective function and identify the most important parameters for prediction. (C) 2014 Elsevier B.V. All rights reserved.