Facet formation and competition in selective epitaxial growth (SEG) of Si on SiO2- masked nanometer growth windows were studied experimentally and theoretically in gas source molecular beam epitaxy. The windows were patterned along [110] on Si(100) substrates. Mass accumulation around the edge of the SEG mesa top surface was observed and believed to be due to the interfacet mass transport from sidewalls to the top surface. Sidewall facet evolution from the initial (311) to (111) was observed. The facet evolution is due to the growth rate anisotropy among (100), (311), and (111). The interfacet mass transport is believed to affect growth rate of each facet and increase the growth rate anisotropy. As a result, it controls the size dependence of SEG growth rate and promotes the facet evolution. A theoretical model has also been developed for modeling the facet competition and evolution. Both the interfacet and intrafacet surface migration processes were taken into account in the model. The simulation results explain our experimental observations and provide an improved understanding of the facet growth on patterned substrates.