Evolutionary innovations, traits that give species access to previously unoccupied niches, may promote speciation and adaptive radiation. Here, we show that such innovations can also result in competitive inferiority and extinction. We present evidence that the modified pharyngeal jaws of cichlid fishes and several marine fish lineages, a classic example of evolutionary innovation, are not universally beneficial. A large-scale analysis of dietary evolution across marine fish lineages reveals that the innovation compromises access to energy-rich predator niches. We show that this competitive inferiority shaped the adaptive radiation of cichlids in Lake Tanganyika and played a pivotal and previously unrecognized role in the mass extinction of cichlid fishes in Lake Victoria after Nile perch invasion.