We report the observation of temporally varying electromagnetic hot spots in plasmonic nanostructures. Changes in the field amplitude, position, and spatial features are induced by embedding plasmonic silver nanorods in the photoresponsive azo-polymer. This polymer undergoes cis-trans isomerization and wormlike transport within resonant optical fields, producing a time-varying local dielectric environment that alters the locations where electromagnetic hot spots are produced. Finite-difference time-domain and Monte Carlo simulations that model the induced field and corresponding material response are presented to aid in the interpretation of the experimental results. Evidence for propagating plasmons induced at the ends of the rods is also presented.