Conjugated polymers have rigid backbones and strong interchain pi-pi interaction in aggregation state, so their morphologies in thin films could be significantly affected by backbone conformation. In this work, taking two highly efficient photovoltaic polymers (PDTBT-TT and PBT4T) as example, we investigated the correlations among the chemical structure of backbone units, thermodynamic stability of backbone conformation, backbone curvature, phase transformation kinetics, and crystalline properties by utilizing quantum chemistry calculation, differential scanning calorimetry, temperature-dependent UV-vis absorption spectroscopy, and X-ray diffraction. The results indicated that compared to PBT4T, PDTBT-TT possesses more rigid backbone, more stable backbone conformation, and curvature, so PDTBT-TT has higher melting and crystallization temperatures but smaller phase transformation enthalpy than PBT4T. Furthermore, polymer solar cells (PSCs) based on these two polymers were fabricated and characterized with a simple conventional device structure. In our measurements, the PSCs based on PBT4T showed a best PCE of 8.38%, which is coincident with the reported results; the PSCs based on PDTBT-TT exhibited a best PCE of 9.18%, indicating PDTBT-TT is a promising photovoltaic polymer material. Overall, the results and discussion in this work suggest a new perspective for molecular design of polymer photovoltaic materials.