Stretch-flangeability evaluated using hole-expansion testing represents the ability of sheet materials to resist edge fracture during complex shape forming. Despite a property imperative for automotive part applications of advanced high-strength steels, factors governing stretch-flangeability are not yet well understood. In this study, the mechanical properties of a selected group of materials with different microstructures were investigated using tensile, fracture toughness, and hole-expansion tests to find the factor governing the stretch-flangeability that is universally applicable to a variety of metallic materials. It was found that the fracture toughness of materials, measured using the fracture initiation energy, is a universal factor governing stretch-flangeability. We verified that fracture toughness is the key factor governing stretch-flangeability, showing that the hole-expansion ratio could be well predicted using finite element analysis associated with a simple ductile damage model, without explicitly taking into account the microstructural complexity of each specimen. This validates the use of the fracture toughness as a key factor of stretch-flangeability.