Metals support surface plasmons at optical wavelengths and have the ability to localize light to subwavelength regions. The field enhancements that occur in these regions set the ultimate limitations on a wide range of nonlinear and quantum optical phenomena. We found that the dominant limiting factor is not the resistive loss of the metal, but rather the intrinsic nonlocality of its dielectric response. A semiclassical model of the electronic response of a metal places strict bounds on the ultimate field enhancement. To demonstrate the accuracy of this model, we studied optical scattering from gold nanoparticles spaced a few angstroms from a gold film. The bounds derived from the models and experiments impose limitations on all nanophotonic systems.