Saving energy is important and necessary for hydraulic excavators. This paper proposed a novel energy regeneration boom system. A variable displacement hydraulic motor and flow control valve were used to regulate the torque and speed of the generator, and to regulate flow through the hydraulic motor. A control strategy was proposed to calculate the optimal displacement of the hydraulic motor. The hydro-mechanical efficiency map and volumetric efficiency map were used to estimate the speed and torque of the generator instead of using speed and torque sensor. The resulting energy regeneration efficiency ranged from 33.8% to 57.4%, which cannot be realized in conventional boom system. Compared with conventional energy regeneration boom system, the improvement of energy regeneration efficiency with the proposed system was 3.2% to 4.1% for low and moderate velocities. A control strategy has been proposed to decrease the setup power of generator and ensure system safety in large load and high velocity mode.