Using the Eshelby equivalent inclusion theory and the Mori-Tanaka method, a new micromechanical model is proposed to predict the tensile modulus of carbon fibers by considering crystallites, amorphous components, and microvoids of the fiber structure. Factors that affect the tensile modulus included the aspect ratio of crystallites, the aspect ratio of microvoids, the volume fraction of crystallites, the volume fraction of microvoids, and the orientation degree of crystallites. To follow the dependence of the tensile modulus of the fibers on microstructure, thirty different types of polyacrylonitrile-based fibers were prepared. The aspect ratios and orientation degrees of crystallites were calculated directly by X-ray diffraction. The aspect ratios and volume fractions of microvoids were obtained by small-angle X-ray scattering. The average tensile modulus of amorphous was estimated by dealing with thirty types of PAN-based fibers. The volume fractions of crystallites were obtained by the micromechanical model. Some relationships are concluded: (1) the tensile modulus increased with increasing volume fractions of crystallites, aspect ratios of crystallites and microvoids, and orientation degree of crystallites; (2) the tensile modulus increased with decreasing volume fractions of microvoids.