The crystal structure and stability of a giant tetragonal phase in electron-doped Pb1-xBixVO3 (x = 0.1, 0.2, and 0.3) and hole-doped Pb1-xNaxO3 (x = 0.1, 0.2, and 0.3) were studied. Electron doping effectively destabilized the tetragonal structure. The c/a ratio, spontaneous polarization, and tetragonal-to-cubic phase transition pressure systematically decreased with increasing Bi3+ substitution. In contrast, hole doping hardly affected the tetragonal distortion and structural stability. We showed that electron doping is an effective way to control the stability of the tetragonal phase of PbVO3 with a 3d(1) electronic configuration.