Based on two-layer multi-objective optimization which is aimed at maximum performance evaluation criterion (PEC) and maximum field synergy number (Fc), the heat transfer and flow characteristics in shell side of helically coiled tube heat exchanger are investigated experimentally and numerically. The sensitivity analysis, univariate analysis and response surface analysis are conducted, in which, the selected working fluids are water and saturated vapor, and the transverse spacing (5), coiled radius (R), heat exchange tube diameter (d) are set as the design variables. The results suggested that d significantly affects f and PEC, while, R and S comparatively small, and for Nu, R is larger, whereas the difference between the three is small. The optimal structure for a Reynolds number from 5000 to 40,000 is that: d = 8 mm, R = 50 mm, S = 20 mm or 21 mm. Compared with the initial structure under the same working condition, the Fc based on the field synergy principle (FSP) indicates the comprehensive performance is improved by 36.84%, and PEC shows 33.42% improvement in the comprehensive performance. The fitting correlations of Nu and f in shell side of helically coiled tube heat exchanger fitted by the numerical simulation data are well consistent with the experimental data. (C) 2019 Elsevier Ltd. All rights reserved.