A new approach is proposed to predict the relationship between geometric arrangement of fibers within the polymer matrix and viscoelastic behavior displayed by unidirectional fiber-reinforced polymers. Based on a quantitative morphology analysis of epoxy/glass fibers composites, a four-phase model taking into account the fiber spatial distribution is developed to rigorously express the reinforcement effect of the polymer matrix over a wide range of fibers and temperatures. Comparisons with other modelings and experiment illustrate the validity of the proposed approach. In addition, changes in the viscoelastic behavior of the epoxy matrix in composite materials resulting from a decrease in the crosslinking degree of such a kind of polymer are, subsequently, assessed through such an approach.