In this study, the simultaneous effects of both silanized coal fly ash (S-CFA) and nano/micro glass fiber (nGF) on fracture toughness and mechanical properties of vinyl ester (VE) resin filled with carbon fiber-based composite materials were investigated. The CFA was treated with (3-trimethoxysilyl) propyl methacrylate to introduce the methacryloxy groups into the surface of CFA, and was confirmed by using FTIR technique. The nGF and S-CFA with different weight ratios were well mixed with VE resin by using of high-speed mechanical stirrer, and ultrasonic technique before using as matrices for fabrication of carbon fiber-based composite materials via sheet molding compound (SMC) method and hot curing processing. Many characteristics of both cured VE resin composites and carbon fiber-based composite were examined such as mechanical properties, fracture toughness, and morphology. The results showed that by adding of both 0.1 wt% nGF and 1 wt% S-CFA into VE resin the tensile strength, tensile modulus, flexural strength, K-IC, impact strength as well as the Mode I interlaminar fracture toughness (G(IC)) of VE composites and carbon fiber based composites get optimum values and increased about 61.39%; 39.83%; 36.21%; 103.1%; 81.79%; 48.61%, respectively when compared with pristine materials. POLYM. ENG. SCI., 59:584-591, 2019. (c) 2018 Society of Plastics Engineers