Propylene random copolymer (PPR)/styrene-ethylene-butylene-styrene block copolymer (SEBS)/compatibilizer/organic-montmorillonite (OMMT) quaternary nanocomposites and PPR/compatibilizer/OMMT ternary nanocomposites were prepared via two-stage melt blending and influences of compatibilizers, maleic anhydride (MA) grafted styrene-ethylene-butylene-styrene copolymer (SEBS-g-MA), poly(octene-co-ethylene) (POE-g-MA), or propylene block copolymers (PPB-g-MA), on rheology and mechanical properties of the nanocomposites were investigated. The results of X-ray diffraction measurement and transmission electron microscopy observation showed that OMMT layers were mainly intercalated in the nanocomposites except for the mainly exfoliated structure in the quaternary nanocomposites using POE-g-MA as compatibilizer. The nanocomposites exhibited pseudo-solid like viscoelasticity in low frequencies and shear-thinning in high shear rates. As far as OMMT dispersion was concerned, POE-g-MA was superior to SEBS-g-MA and PPB-g-MA, which gives rise to the highest viscosities in both the ternary and quaternary nanocomposites. The quaternary nanocomposites containing POE-g-MA were endowed with balanced toughness and rigidity. It was suggested that a suitable combination of compatibilizer and SEBS was an essentially important factor for adjusting the OMMT dispersion and distribution, the rheological and mechanical performances of the nanocomposites. (c) 2012 Wiley Periodicals, Inc. J. Appl. Polym. Sci., 2013