Polymer modifiers have been used to improve the performances of asphalt binders in pavement engineering. The modifying effect of polymers on asphalt is largely dependent on the morphological characteristics of polymer-modified asphalt. The morphologies of polymer-modified asphalt are composed of a polymer-rich phase, a asphaltene-rich phase, and the interphase between the two phases. Interfacial interactions importantly contribute to the morphology but are commonly overlooked. In this study, carbon nanotubes (CNTs) were selected to improve the interfacial interactions of polymer-modified asphalt. Fluorescence microscopy (FM), scanning electron microscopy (SEM), micro-Raman spectroscopy (MRS), and molecular dynamics (MD) simulation were used to capture the characteristics of the interphase and polymer-rich phase. CNTs-polymer-modified asphalt involves stronger intermolecular forces than those in asphalt-modified by only styrene butadiene styrene (SBS) or CNTs. This discrepancy highlights the intensified interfacial interaction in the former material. Raman peak and MD findings reveal that the C=C of CNTs interacted with the alkanes and aromatic hydrocarbons of asphalt. SBS were entwined or surrounded with CNTs through the pi-pi conjugation of the benzene rings of the two components. Consequently, a synergistic effect enhanced the intermolecular force between SBS and CNTs in the interphase. SEM results indicated that CNTs were enriched in the interphase, enhancing mechanical anchorage between the polymer and asphalt. As a result, CNTs increased the roughness of the interphase and produced a prominent cage construction of polymer-rich phase. Moreover, the observed pullout behaviors of CNTs alleviated interfacial failure. FM images displayed that CNTs enhanced the swelling degree of the polymer-rich phase. This effect was realized because CNTs served as a tunnel for transporting saturates, aromatics, and small resin molecules, as shown by molecular dynamics MD analysis. This work revealed the importance of the interfacial interactions on the micromorphologies of polymer-modified asphalt.