We propose a microscopic theory of light-induced deformation of side-chain azobenzene polymers taking into account the internal structure of polymer chains. Our theory is based on the fact that interaction of chromophores with the polarized light leads to the orientation anisotropy of azobenzene macromolecules which is accompanied by the appearance of mechanical stress. It is the first microscopic theory which provides the value of the light-induced stress larger than the yield stress. This result explains a possibility for the inscription of surface relief gratings in glassy side-chain azobenzene polymers. For some chemical architectures, elongation of a sample demonstrates a nonmonotonic behavior with the light intensity and can change its sign (a stretched sample starts to be uniaxially compressed), in agreement with experiments. Using a viscoplastic approach, we show that the irreversible strain of a sample, which remains after the light is switched off, decreases with increasing temperature and can disappear at certain temperature below the glass transition temperature. This theoretical prediction is also confirmed by recent experiments.