Based on theories of thermodynamics, the energy equation in terms of entransy in heat transfer process is introduced, which not only describes the change of entransy, but also defines the entransy consumption rate. According to the regularity of entransy change in heat transfer process and the effect of entransy consumption rate on the irreversibility of heat transfer process, it can be found that entransy is a state variable, from which a new expression for the second law of thermodynamics is presented. Then by setting entransy consumption rate and power consumption rate as optimization objective and constraint condition for each other, the Lagrange conditional extremum principle is used to deduce momentum equation, constraint equation and boundary condition for optimizing flow field of convective heat transfer, which are applied to simulate convective heat transfer coupling with energy equation in an enclosed cavity. Through the numerical simulation, the optimized flow field under different constraint conditions is obtained, which shows that the principle of minimum entransy consumption is more suitable than the principle of minimum entropy generation for optimizing convective heat transfer process. (C) 2011 Published by Elsevier Ltd.