Black phosphorus quantum dots (BPQDs), as 2D van der Waals crystals, possess remarkable electronic and optoelectronic properties. Their excellent ultra-high mobility and thickness-dependent tunable direct bandgap can meet well the needs of polymer solar cells (PSCs) for adjustable energy levels in different active layer systems. The formation of better energy alignment in the devices is significantly beneficial to the charge transfer, exciton dissociation and reduce charge recombination, which can lead to improved power conversion efficiency (PCE). Herein, it is introduced to modify hole transport layer (HTL) as interfacial modifier (IM) in fullerene and non-fullerene PSCs. The cascade band structure is formed successfully between the anode and polymer donor. In PTB7-Th:PC71BM system, PCE is increased from 8.12% to 9.11%. In PM6:IT-4F system, PCE is enhanced from 11.65% to 12.81%. BPQDs are utilized as HTL IM in conventional device in our study, which can open up a promising route for photovoltaic research.