Gauge fields in condensed matter physics give rise to nonreciprocal and topological transport phenomena and exotic electronic states(1). Nanomechanical systems are applied as sensors and in signal processing, and feature strong nonlinearities. Gauge potentials acting on such systems could induce quantum Hall physics for phonons at the nanoscale. Here, we demonstrate a magnetic gauge field for nanomechanical vibrations in a scalable, on-chip optomechanical system. We induce the gauge field through multi-mode optomechanical interactions, which have been proposed as a resource for the necessary breaking of time-reversal symmetry(2-4). In a dynamically modulated nanophotonic system, we observe how radiation pressure forces mediate phonon transport between resonators of different frequencies. The resulting controllable interaction, which is characterized by a high rate and nonreciprocal phase, mimics the Aharonov-Bohm effect(5). We show that the introduced scheme does not require high-quality cavities, such that it allows exploring topological acoustic phases in many-mode systems resilient to realistic disorder. Gauge fields in condensed matter give rise to nonreciprocal transport and topological non-trivial states. In an on-chip experiment, multi-mode optomechanical interactions generate a magnetic gauge field for nanomechanical motion and yield phonon transport with a nonreciprocal phase.