Liquid separation condenser (LSC) is a newly developed fin-and-tube condenser. It is superior to other traditional condenser because of its relatively low pressure drop without reducing heat transfer coefficient. It is suitable to be used in the organic Rankine cycle. However, the design of LSC is a very complex and time consuming task base on traditional methods. In this study, a new modelling, simulation and optimization methodology for LSC design is developed. A pass-by-pass tube side modelling method is proposed. An equivalent heat transfer coefficient and total pressure drop are defined, modelled and validated. Then, a mathematical model containing multiple continuous and discrete variables for the optimal design of LSC is developed. The main continuous variables are tube length, pressure drops, fluid velocity, air velocity, heat transfer area, and outlet air temperature. The main discrete variables are the type selection of finned-tubes, number of passes, number of tubes per pass, fin number per unit tube length. The objective function is the minimization of the total cost, which is the best trade-off between the heat transfer coefficient and the pressure drop. The resulting model is a non-convex mixed integer non-linear programming (MINLP) model. A solving strategy that integrates model relaxation, solver selection, and tube-pass scheme initialization is proposed. Two case studies are elaborated to test the effectiveness of the proposed methodology. Comparison of the optimization results with base case reveals that the proposed methodology can be successfully applied to the design optimization of LSC. The influences of pass number, fin number, tube-fin type, tube number, and investment cost on the optimization results are also discussed. (C) 2015 Elsevier Ltd. All rights reserved.