A novel reactive phosphorusnitrogen-containing monomer, N-(2-(5,5-dimethyl-1,3,2-dioxaphosphinyl-2-ylamino)ethyl)-acrylamide (DPEAA), was synthesize and characterized. Flame retardant poly(methyl methacrylate)/organic-modified montmorillonite (PMMA-DPEAA/OMMT) nanocomposites were prepared by in situ polymerization by incorporating methyl methacrylate, DPEAA, and OMMT. The results from X-ray diffraction and transmission electron microscopy (TEM) showed that exfoliated PMMA-DPEAA/OMMT nanocomposites were formed. Thermal stability and flammability properties were investigated by thermogravimetric analysis, cone calorimeter, and limiting oxygen index (LOI) tests. The synergistic effect of DPEAA and montmorillonite improved thermal stability and reduced significantly the flammability [including peak heat release rates (PHRR), total heat release, average mass loss rate, etc.]. The PHRR of PMMA-DPEAA/OMMT was reduced by about 40% compared with pure PMMA. The LOI value of PMMA-DPEAA/OMMT reached 27.3%. The morphology and composition of residues generated after cone calorimeter tests were investigated by scanning electronic microscopy (SEM), TEM, and energy dispersive X-ray (EDX). The SEM and TEM images showed that a compact, dense, and uniform intumescent char was formed for PMMA-DPEAA/OMMT nanocomposites after combustion. The results of EDX confirmed that the carbon content of the char for PMMA-DPEAA/OMMT nanocomposites increased obviously by the synergistic effect of DPEAA and montmorillonite. (c) 2011 Wiley Periodicals, Inc. J Appl Polym Sci, 2011