Effects of different electrolytes on the surface structure and interfacial property with epoxy resin of PAN-based carbon fiber are evaluated, which is used to find out the key factor that affects the improvement in the interfacial adhesion. It is found that the carbon fiber treated in H2SO4 electrolyte owns the most abundant oxidative groups but bonds relatively weak with epoxy resin by the maximum interlaminar shear strength of 60 MPa. The carbon fiber oxidized in NaOH electrolyte and then restored in HCl solution owns less oxidative groups but shows a strongest bonding with epoxy resin by the maximum interlaminar shear strength of 69 MPa. We conclude this phenomenon is mainly ascribed to the graphene layers covering on the fiber surface that are formed during carbonization and connect weakly with the substrate of carbon fiber, thus called weak layers. The weak layers can be peeled off effectively in NaOH electrolyte but not in H2SO4 electrolyte. An attempt for modification is tried by removing the weak layers in NaOH electrolyte and adding more oxygen groups in H2SO4 electrolyte. The interlaminar shear strength of the fiber with epoxy resin is not increased as expected; however, it shows oxidation in H2SO4 electrolyte does not generate weak layers on carbon fiber surface.