We present a study on macroscale friction of polyethylene nanofibrillar arrays against patterned rough surfaces with various asperity heights, spacings, and area fractions. These surfaces are prepared by utilizing colloidal lithography and silica evaporation, which allows the independent control of geometric parameters. While the nanofiber arrays exhibit high friction on a smooth surface, much lower friction is observed when the asperity height becomes larger than can be compensated by fiber compliance, or when the asperity spacing becomes small enough to prevent fiber penetration for contact. The observed behavior is discussed with simple mechanical models and summarized to provide some criteria to maintain high friction against rough surfaces.