Because of the inherent characteristics of the thiol-ene step growth mechanism in preparation of thiol-ene photopolymer clay nanocomposites, the ratio between thiol and ene functional groups at and near the organoclay surfaces may have a significant effect on the polymerization behavior. This study investigates the influence of monomer composition and the type of polymerizable organoclay on thiol-acrylate photopolymerization behavior in preparation of photocurable clay nanocomposite systems. To this end, two types of polymerizable organoclays with acrylate or thiol functional group on the clay surfaces were compared in monomer compositions with different polarity and functionality. Real-time infrared spectroscopy was used to characterize polymerization behavior in conjunction with photo-DSC. The degree of clay exfoliation was evaluated using small angle X-ray scattering and correlated with photopolymerization behavior. Higher chemical compatibility of components induced enhanced clay exfoliation resulting in increase in photopolymerization rate. By affecting the stoichiometric ratio of functional groups in the clay gallery, thiolated organoclays enhance thiol-ene reaction, whereas acrylated organoclays encourage acrylate homopolymerization. In addition, inducing more propagating thiyl radicals on the organoclay surfaces by increasing functionality of thiol monomer also facilitates thiol-ene copolymerization, whereas the increase of acrylate functionality reduces final thiol conversion. (C) 2010 Wiley Periodicals, Inc. J Polym Sci Part A: Polym Chem 49: 465-475, 2011