In this study, Response Surface Methodology (RSM) is applied to find the optimal profile of wavy-wall for an enclosure to study the nanoparticles treatment around the heated cylinder. Based on previous study, an enclosure with two side wavy-walls and a heated cylinder located in its center is considered and natural convection heat transfer of nanofluids is analyzed by finite element method (FEM). To reduce the number of numerical runs, nine different profiles are chosen by central composite design (CCD) and the results of Nusselt numbers for both wavy-wall and cylinder wall are extracted. To reach the purpose of maximum Nusselt number, the optimized profile is introduced and finally the effect of heated cylinder diameter on the heat transfer in the optimized case in discussed. Results show that increasing the cylinder diameter up to 1.0 can enhance the heat transfer while more than this, make reduction in Nusselt number. (C) 2016 The Society of Powder Technology Japan. Published by Elsevier B.V.