The coupling between the magnetic and electric dipoles in multiferroic and magnetoelectric materials holds promise for conceptually novel electronic devices(1-3). This calls for the development of local probes of the magnetoelectric response, which is strongly affected by defects in magnetic and ferroelectric ground states. For example, multiferroic hexagonal rare earth manganites exhibit a dense network of boundaries between six degenerate states of their crystal lattice, which are locked to both ferroelectric and magnetic domain walls. Here we present the application of a magnetoelectric force microscopy technique that combines magnetic force microscopy with in situ modulating high electric fields. This method allows us to image the magnetoelectric response of the domain patterns in hexagonal manganites directly. We find that this response changes sign at each structural domain wall, a result that is corroborated by symmetry analysis and phenomenological modelling(4), and provides compelling evidence for a lattice-mediated magnetoelectric coupling. The direct visualization of magnetoelectric domains at mesoscopic scales opens up explorations of emergent phenomena in multifunctional materials with multiple coupled orders.