Insulator-based dielectrophoresis (iDEP) is a powerful technique for separation and manipulation of bioparticles. In recent years, iDEP designs using arrays of insulating posts have shown promising results toward reaching high-efficiency bioparticle manipulation. Joule heating (JH) and electrothermal (ET) flows have been observed in iDEP microdevices and significantly affecting their performances. In this research, we utilize mathematical modeling to study, iDEP technique and the effects of JH and ET flow on device performance and propose a separation scenario for selective trapping of circulating tumor cells (CTCs). A robust numerical model is developed to calculate the distribution of electric and fluid flow fields in the presence of JH and ET flow, and predict the cells' trajectory inside the system. Our results indicate that JH not only induces temperature rise in the system, but also may alter the design iDEP separation scenario by inducing ET vortices that affect the cell's trajectory. To investigate the impact of JH-induced ET flow characteristics and vortex generation on separation efficiency, we introduce a dimensionless force ratio encompassing the effects of electrical field, drag forces, JH, and ET flow. Interestingly, it was found that ET flows can be used to significantly enhance the separation efficiency, even in higher inlet flow rates. Lastly, the effect of post geometry has been discussed.