The hybrid material of EP-POSS mixture was synthesized by the hydrolysis and condensation of (c-glycidoxypropyl) trimethoxysilane. A series of binary systems of EP-POSS/epoxy blends, epoxy resin modified by silica nanoparticles (SiO2/epoxy), and ternary system of SiO2/EP-POSS/epoxy nanocomposite were prepared. The dispersion of SiO2 in the matrices was evidenced by transmission electron micrograph, and the mechanical properties, that is, flexural strength, flexural modulus, and impact strength were examined for EP-POSS/epoxy blends, SiO2/epoxy, and SiO2/EP-POSS/epoxy, respectively. The fractured surface of the impact samples was observed by scanning electron micrograph. Thermogravimetry analysis were applied to investigate the different thermal stabilities of the binary system and ternary system by introducing EP-POSS and SiO2 to epoxy resin. The results showed that the impact strength, flexural strength, and modulus of the SiO2/EP-POSS/epoxy system increased around by 57.9, 14.1, and 44.0% compared with the pure epoxy resin, Ti, Tmax and the residues of the ternary system were 387 degrees C, 426 degrees C, and 25.2%, increased remarkably by 20 degrees C, 11 degrees C and 101.6% in contrast to the pure epoxy resin, which was also higher than the binary systems. (C) 2013 Wiley Periodicals, Inc.