A novel poly(methyl methacrylate) (PMMA)-based nanocomposite combined with a reactive flame retardant (tetramethyl (3-(triethoxysilyl) propyl-azanediyl) bis(methylene) diphosphonate (TMSAP)) and organo-modified layered aluminophosphate (OLAP) was synthesized by the sol gel method. The structure of the nanocomposite achieves maximal integration of both merits of each component, such as slime cross-linking function and dehydration charring effect of TMSAP and physical barrier effect and catalytic-charring effect of OLAP, which promotes cross-linked network formation, improves the quality and quantity of char, and inhibits the heat, oxygen, and mass transfer, leading to significant enhancements of scratch resistance, thermal stability, and flame retardancy. Compared to PMMA, the nanocomposites maintain high transparency and exhibit increased shore hardness by 90%, glass transition temperature by 13 degrees C, and half degradation temperature by 105.1 degrees C; in addition, peak heat released rate decreased by 59.1%. This work demonstrates the simultaneous enhancement of flame retardancy, thermal properties, and mechanical performance of polymers.