Rough nanostructured surfaces can enhance evaporation heat transfer. Most studies artificially optimized the geometry and size during the design of nanostructured surfaces. Instead of the empirical design of nanostructured surfaces, this paper proposes a mathematical optimization method of the grooved nanostructured surface design. This method is inspired by the molecular dynamics simulations of grooved nanostructured surfaces. The results show that the heat transfer performance exhibits a positive correlation with the defined sectional area of the grooved nanostructured surface; thus, this method is developed to convert the maximum heat transfer and evaporation rate to a mathematical conditional extremum solution. The mathematical description of the optimization method is to solve the surface structure with the maximum sectional area when the heat transfer area is constant. Comparing the molecular dynamics (MD) simulation results of the optimal surface and the existing ones under the same simulation conditions indicates that the optimal surface has the best heat transfer performance compared with the other ones. Additionally, discussions on the types of grooved nanostructured surfaces, the materials of solid and liquid, and the wettability of grooved surfaces verify the generality of the calculation results and the optimization method. The explanation of the method is that different nanostructured surfaces have a similar potential energy per liquid atom, which affects the latent heat of the evaporation process. However, the maximum sectional area corresponds to the minimum interfacial thermal resistance and the maximum interaction energy per unit area, which will enhance the heat transfer at the solid-liquid interface. Moreover, a nanostructured surface with the maximum sectional area also obtains the maximum area of the liquid-vapor interface and thus enhances the evaporation heat transfer process.