The thermoelectric (TE) device enables a conversion interface between the heat transfer and the electricity domain. Specifically, it can operate bi-directionally - Heat can be converted to electricity via the thermoelectric generator (TEG) effect and conversely, electricity can be converted to heat via the thermoelectric cooling (TEC) effect. As to date, most publications deal with the optimization of the TE device either in terms of the TEG performance or in terms of the TEC performance. This paper brings the two design approaches together to form a multi-objective optimization of the TE device. By performing such an optimization, the relationships between the TEG and TEC performance criterion are better established with key factors including any potential trade-off situations and possibilities for optimal design towards both objectives. Moreover, the key parameters that have the strongest influence on these criteria and the related trends are identified. Optimization results indicate that there is a strong correlation between the TEG performance and system mass but the correlation is less effective from the TEC perspective. On the other hand, the constraint requirements on the TEG design is found to have strong influence on the optimality of the TEC solutions.