Flexible ablative materials play a key role in thermal protection systems to protect space vehicles during hypersonic flight missions. In this work, epoxy resin modified liquid silicone rubber was adopted as elastomeric matrix. Two inorganic fibers (carbon fibers (CF) and quartz fibers (QF)) and two organic fibers (aramid fibers (AF) and poly (p-phenylene benzobisoxazole) fibers (PBO)) were chosen as functional fillers. The ablation resistance and thermal insulation properties were evaluated using oxyacetylene torch test. Microstructure and phase composition of the char layer was fully characterized to investigate the ablation mechanism. SEM observations revealed that the fibers are well dispersed and integrated in the matrix. TGA analysis indicated that the inorganic fibers possess significantly higher thermal stability than organic fibers. In addition, a molten silica film can be formed on the material surface, which can exert a better thermal protection effect on the matrix. Furthermore, the ablation test shows that the linear ablation rate decreases first and then increases with an increase of fiber content. This work provides basic data to guide optimal selection of fibrous ablatives to enhance ablation performance.