Understanding the CO2 transfer processes in shale oil reservoir is essential for the development of shale oil reservoir by CO2 injection. Although many mathematical models have been proposed to evaluate CO2 diffusion in oil-saturated porous media, very few models consider the effect of kerogen on the CO2 transfer processes in shale oil reservoir, and research studies systematically combining a mathematical model history-matched to experimental data is also rare. To describe these processes involved in the measurements, a mathematical model and numerical solutions were derived. In solving the mathematical model, the adsorption and dissolution of CO2 in shale and oil-phase swelling caused by CO2 dissolution were considered. From the fitted experimental results of the numerical method, the effective diffusion coefficients of CO2 in shale and tight sandstone were estimated at different pressures and the effective diffusion coefficients in shale were larger than those in tight sandstone at high pressures (> 9.1 MPa). The sensitivity analyses for the model reveal that the effective diffusion coefficient, oil-swelling factor, and the adsorption and dissolution quantity of CO2 determine both the pressure decay and equilibrium time of the gas transport processes. The analysis and comparison of the modeling results and experimental results reveal that the effect of CO2 adsorption and dissolution in shale was stronger than the oil swelling. Finally, quantities of the free CO2 and the adsorption and dissolution CO2 in shale sample were calculated by the given model. The effective diffusion coefficients measured through the proposed method already incorporated the characteristics of shale and thus can be used directly in modeling CO2 transport in shale oil reservoir. The study also indicates that the geological storage of CO2 in shale oil reservoir is one of the mitigation measures for the greenhouse effect.