A liquid carboxyl-terminated butadiene-acrylonitrile copolymer (CTBN) and SiO2 particles in nanosize were used to modify epoxy, and binary CTBN/epoxy composites and ternary CTBN/SiO2/epoxy composites were prepared using piperidine as curing agent. The morphologies of the composites were observed by scanning electron microscope (SEM) and transmission electron microscope (TEM), and it is indicated that the size of CTBN particles increases with CTBN content in the binary composites, however, the CTBN particle size decreases with the content of nanosilica in the ternary composites. The effects of CTBN and nanosilica particles on the mechanical and fracture toughness of the composites were also investigated, it is shown that the tensile mechanical properties of the binary CTBN-modified epoxy composites can be further improved by addition of nanosilica particles, moreover, obvious improvement in fracture toughness of epoxy can be achieved by hybridization of liquid CTBN rubber and nanosilica particles. The morphologies of the fractured surface of the composites in compact tension tests were explored attentively by field emission SEM (FE-SEM), it is found that different zones (pre-crack, stable crack propagation, and fast crack zones) on the fractured surface can be obviously discriminated, and the toughening mechanism is mainly related to the stable crack propagation zone. The cavitation of the rubber particles and subsequent void growth by matrix shear deformation are the main toughening mechanisms in both binary and ternary composites.