All-polymer solar cells (All-PSCs) have attracted a lot of attention in the recent years due to broad absorption spectra, adjustable chemical structures, high morphological stability, better donor-acceptor compatibility and better mechanical durability. This review summarizes the recent development of perylene diimides (PDI)-based electron acceptors used in all-PSCs because these polymers are promising candidates. The photovoltaic performances of non-fullerene polymer acceptors were compared, and the structure-performance relationship was discussed when the same donor material was used.