Flow instability in three entangled polymer systems including a 10 wt% 1,4-polybutadiene (PBD) solution, an 11.4 wt% polyisobutylene (PBD) solution, and a long chain branched polyethylene melt (LD 146) was investigated in both stress-controlled and rate-controlled experiments in the cone-plate geometry. It was found that flow instability occurred for experiments in both rate- and stress-controlled modes. The effects of cone angle or rim gap and shearing time on flow instability were studied. The smaller cone angle and shorter shearing time delay (in terms of stress or shear rate) the occurrence of severe instability and mass loss of the PBD solution but not for the PIB. Our data are consistent with the dramatic shear rate jump for the flow curve constructed from the stress-controlled experiments being associated with mass loss after the severe instabilities. We also find that the Cox-Merz representation gives a powerful tool for investigation of flow instability. Finally, another interesting result in this work is that it seems that the stress overshoot can be related to the onset of flow instability in the present system.