To improve air efficiency in the combustion chamber, a new separated swirl combustion system (SSCS) is proposed in this study. The combustion performance and emission characteristics were analyzed using three-dimensional simulation. A sensitivity analysis of indicated power and soot emission by chamber geometries was carried out. Based on the results of the sensitivity analysis, the chamber geometries were optimized. Then, the combustion and emission performance of the SSCS with optimized chamber geometries under different speeds was tested and compared with a double swirl combustion system (DSCS) in a single-cylinder diesel engine. The simulation results show that because the angle of the inner chamber affects the fuel distribution in the inner and outer chamber, it has the greatest influence on the SSCS combustion performance. The depth of the separated chamber is the second most important optimization parameter, followed by the diameter of the outer chamber, the separated chamber diameter, the angle of the outer chamber and finally the volume ratio of the inner and outer chamber. By comparing the combustion performance, fuel/air mixture, velocity distribution and soot emission in the cylinder of different SSCS chamber geometries, the optimized chamber geometries were determined. Then, the combustion and emission performance of the SSCS with optimized chamber geometries was compared with a DSCS combustion system. The results show that the brake-specific fuel consumption (BSFC) was lower and the indicated thermal efficiency was higher in the SSCS than in the DSCS at various engine speeds. The results of this study are of great significance for the optimization of chamber geometries as well as the improvement of fuel/air mixture and combustion performance.