Hyperbranched thermosetting resin is the best additive for toughening and reinforcing linear thermosetting resin. A novel low-viscosity unsaturated hyperbranched polyester resin (UHPR) is synthesized by the reaction between maleic anhydride monoisooctyl alcohol ester and a hydroxyl-ended hyperbranched polyester resin (HPR) prepared from phthalic anhydride (PA) and trimethylolpropane (TMP). The structure of HPR is characterized by FT-IR and (HNMR)-H-1, and its degree of branching is deduced by comparing with the (HNMR)-H-1 spectrum of a model compound. The molecular weights of HPR and UHPR are determined by theoretical calculation and MALDI-TOF MS measurements. This low-viscosity (< 10 000 cP) novel UHPR can be applied in the field of environment-friendly coatings and in toughening and reinforcing linear unsaturated polymers.