The mechanical behavior of acrylonitrile butadiene rubber (NBR) -organo modified layered silicate was modeled using Design of Experiments (DoE). Response surface methodology (RSM), a DoE tool was used to optimize the formulations for optimal performance of the nanocomposites. A Box-Behnken design with three factors and three levels was used to model the relationship between mechanical properties and levels of ingredients. The factors studied for the design are silica content, nanoclay loading and vulcanization system. The nanocomposites were evaluated for tensile strength, modulus, elongation at break and hardness. The effect of heat aging on mechanical properties was also studied. The predicted properties of the nanocomposites are in good agreement with the experimental results, which confirmed the prognostic ability of response surface methodology. The model equations were used to generate response surfaces and contour plots to study the interaction between the variables. The contour plots were overlaid within the applied constraints to identify the required combination of variables that gives the optimum performance for the nanocomposites. (C) 2010 Wiley Periodicals, Inc. J Appl Polym Sci 118: 3300-3310, 2010