Building energy performance simulation tools such as EnergyPlus, Ecotect, and eQuest are widely used to model energy performance of existing buildings and assess retrofit alternatives. Nevertheless, predictions from simulations typically deviate from actual measurements. Monitoring actual performance and measuring deviations from simulated data in 3D can help improve simulation accuracy through model calibrations, and in turn facilitate identification of energy performance problem. To do that, this paper presents Energy Performance Augmented Reality (EPAR) modeling that leverages collections of unordered digital and thermal imagery, in addition to computational fluid dynamics (CFD) models. First, users collect large numbers of digital and thermal imagery from the building under inspection using a single thermal camera. Through an image-based reconstruction pipeline, actual 3D spatio-thermal models are automatically generated and are superimposed with expected building energy performance models generated using CFD analysis through a user-driven process. The outcomes are EPAR models which visualize actual and expected models in a common 3D environment. Within the EPAR models, actual measurements and simulated results can be systematically compared and analyzed. The method is validated on typical residential and instructional buildings. The results demonstrate that EPAR models facilitate calibration of building energy performance models and support detection and analysis of building performance deviations. (C) 2013 Published by Elsevier B.V.