Fe3O4 nanoparticles were modified by n-octadecyltrimethoxysilane (C18TMS) and 3-trimethoxysilylpropylmethacrylate (MPS). The modified Fe3O4 nanoparticles were used to prepare Fe3O4/polystyrene composite particles by miniemulsion polymerization. The effect of surface modification of Fe3O4 on the preparation of Fe3O4/polystyrene composite particles was investigated by transmission electron microscopy, Fourier transform infrared spectrophotometer (FT-IR), contact angle, and vibrating sample magnetometer (VSM). It was found that C18TMS modified Fe3O4 nanoparticles with high hydrophobic property lead to the negative effect on the preparation of the Fe3O4/polystyrene composite particles. The obtained composite particles exhibited asymmetric phase-separated structure and wide size distribution. Furthermore, un-encapsulated Fe3O4 were found in composite particles solution. MPS modified Fe3O4 nanoparticles showed poor hydrophobic properties and resulted in the obtained Fe3O4/polystyrene composite particles with regular morphology and narrow size distribution because the ended C=C of MPS on the surface of Fe3O4 nanoparticles could copolymerize with styrene which weakened the phase separation distinctly.