The reaction between unsaturated polyester (UP) resin and styrene (St) is a heterogeneous free-radical chain-growth cross-linking copolymerization. Curing of the UP/St system in the presence of organically-modified nanoday was studied by differential scanning calorimetry (DSC) and Fourier transform infrared (FTIR) spectroscopy. A mechanistic kinetic model based on the free radical copolymerization mechanism was developed to simulate the reaction rate and conversion profiles of UP/St resin mixtures with various nanoclay contents cured at low temperatures. The model parameters were determined from several DSC experiments under isothermal conditions. The model, in conjunction with heat transfer analysis, was able to successfully predict the temperature and conversion profiles during curing in two vacuum-infusion liquid composite molding (e.g., the Seemann composite resin infusion molding process [SCRIMP] experiments. The presence of nanoclay particles enhanced the tensile modulus, but reduced the tensile strength of the UP nanocomposites. The fracture toughness parameter K-IC was improved by 30% with the addition of 5 wt% nanoclay. The system with mixed nanoclay and calcium carbonate was found to possess the highest K-IC without sacrificing the tensile strength. (c) 2005 Society Plastics Engineers.