The optimal design and operation of an integrated building energy system consisting of the renewable energy technologies such as ground source heat pumps (GSHPs) and solar thermal collectors, etc., is an important problem to be addressed. This paper describes a methodology for the optimization of a building energy system including a detailed thermal model of a borehole heat exchanger based GSHP. The novelty of this model is that it enables the study of dynamic temperature changes within the ground during operation. Furthermore, a model of solar thermal collectors is also included, which enables the study of solar regeneration of the ground in the short and long-term. Additionally, seven scenarios of building envelope retrofit are evaluated alongside optimal system design solutions. The methodology uses a bi-level multi-objective optimization approach, which consists of a genetic algorithm at the design level, and a mixed integer linear program at the operation level, in order to minimise the total costs and CO2 emissions. The methodology is applied to a single-family residential building in Zurich, Switzerland, in order to demonstrate its application and analyse the design and operation of the system, with special attention to the GSHP. The results indicate that in the short-term, the ground temperature reduces considerably, to almost 5 degrees C as compared to the initial temperature of 11.5 degrees C. Furthermore, solar regeneration due to excess heat in summer increases the temperature back above initial temperature. However, due to due to insufficient regeneration in the long-term, the ground temperature drops consistently to almost 4 degrees C at the end of 20 years of operation. On the demand-side, window retrofitting results in a 27.3% reduction in the total CO2 emissions at almost no additional costs. Retrofitting the whole building including windows, walls, roofs, and floors, is a CO2 optimal solution however, performs worst in terms of cost optimality.