It is demonstrated that light polarization can be used to control photothermal effect-based shape-memory polymers (SMPs). Gold nanorods (AuNRs) are embedded in poly(vinyl alcohol) (PVA) and aligned by stretching the composite film. By changing the polarization direction of the incident laser at 785 nm with respect to the film stretching direction, the magnitude of the longitudinal surface plasmon resonance of AuNRs can be varied continuously, which determines the amount of heat generated upon laser exposure and thus the local temperature rise in the composite relative to the glass transition of the PVA matrix. Consequently, the temporary-to-permanent shape recovery process of the composite can be made to occur to different extents by tuning the polarization of laser while keeping all other conditions unchanged. This finding enhances the toolbox for controlling light-triggered SMPs.