Bio-oil tends to polymerize on heating, and understanding the mechanism by which it occurs is of importance for developing the methods for enhancing the polymerization of bio-oil to produce carbon materials or to minimize the polymerization in hydrotreatment. This study mainly investigated the potential cross-polymerization between the typical sugars (xylose and glucose) and phenolics (vanillin and guaiacol) in bio-oil with water as the reaction media in the presence/absence of H2SO4. The results showed that vanillin displayed a high tendency toward polymerization in both the presence/absence of H2SO4, originating from its carbonyl functionality, while guaiacol was quite stable. Nevertheless, in the coexistence of xylose or glucose, guaiacol or vanillin could effectively cross-polymerize with the sugars, resulting in the higher conversion and higher yields of the solid product. During the cross-polymerization, the dehydration of the sugars did not occur substantially, resulting in the high oxygen content and the low thermal stability of the resulting polymers. In addition, in the H2SO4-catalyzed cross-polymerization between vanillin and sugars, the carbonyl functionality was consumed while it was retained in cross-polymerization of guaiacol and sugars, indicating the distinct mechanism for the polymerization.