A 0.4C-2Si-1Cr-1Mo steel bar with an ultrafine-elongated grain (UFEG) structures was produced by multi-pass warm caliber rolling. The test sample was machined from the rolled bar with 0A degrees, 45A degrees, and 90A degrees rotation along the rolling direction, and a static three-point bending test was conducted at ambient temperature. The toughness anisotropy on the steel with UFEG structures were studied, including the crack propagation on the basis of the microstructural features. The strength and toughness decreased with an increase in the rotation angle along the rolling direction. The toughness decreased drastically, compared to the strength. The notch orientation dependence on toughness is due to differences in the spatial distribution of weak sites such as {100} cleavage planes and boundaries of elongated grains. For the toughness design in ultrafine-grained materials, it is essential to understand the spatial distribution of these weak sites as well as the grain size.