The potential of nanoclay organic modifiers to induce plasticizing effects in resin and coatings systems was studied. In previous work, it was found that while low amounts of incorporation of organomodified clays significantly improved the physical and mechanical properties of a ultraviolet (UV)-curable nanocomposite, further increasing the organomodified clay content could result in the reduction of properties. To investigate the potential impact of the organic modifier composition and concentration on polymer properties, a series of experiments were carried out using only the organic modifier. Methyl, tallow, bis-2-hydroxyethyl ammonium (MTEtOH), the organic modifier used in montmorillonite clay Cloisite (R) 30B, was dispersed with precursor polyester oligomers at 110 wt % through an in situ synthesis process and via sonication, and UV-curable coatings were prepared from these MTEtOH-containing resins. The organic modifier cetyltrimethylammonium bromide (CTAB) was also studied to examine the impact of the organic modifier structure. According to differential scanning calorimetry, small decreases in the glass transition temperatures (Tg) of the MTEtOH-containing polyesters were observed, but CTAB-containing polyesters had small Tg increases. Polyester molecular weight and viscosity were also affected by both the structure of the organic modifier as well as its concentration. The mechanical performance of the UV-curable coatings diminished with increased MTEtOH concentration for the films containing the organic modifier compared to a control film. Furthermore, the crosslink density was found to reduce approximate to 50% with increased MTEtOH loading into the UV-curable films. The cure characteristics, thermal stability, and optical clarity were also studied. (c) 2012 Wiley Periodicals, Inc. J. Appl. Polym. Sci., 2013