A wide range of process parameters regulate the final morphology achieved in layered silicate based polymer nanocomposites. This study deals with the optimization of process variables to improve the matrix formulation. A three-factor, three-level Box-Behnken design with compatibilizer concentration (X(1)), clay concentration (X(2)), and screw speed (X(3)) as the independent variables were selected for the study. The dependent variable was mechanical property of the final nanocomposites. Maleic anhydride grafted polypropylene (PP-g-MA) compatibilizer and organoclay (Cloisite 15A) was melt blended with polypropylene separately in a corotating twin screw extruder. The clay was modified with fluorescent dye Nile Blue A Perchlorate (NB) and the adsorbed dye content in the clay gallery was estimated by using UV-spectrophotometric method. The Minitab-15 software was used for analysis of the results obtained. Optimum compositions for better dispersion were achieved from contour plots and response surface methodology. It was supported by a unique fluorescence spectrophotometry along with transmission electron microscopy and X-ray diffraction technique. An intensity ratio close to unity showed a better exfoliated morphology. (C) 2011 Wiley Periodicals, Inc. J Appl Polym Sci 123: 2042-2051, 2012