The interaction of small gold clusters (Au-n, n = 1-4, 20) and a gold monolayer with the MgO (100) surface surrounding a neutral oxygen vacancy (F-s center) is investigated using density-functional (DF) calculations. It is found that the presence of the defect modifies the interaction of gold not only with the vacancy itself, but also with the oxygen and magnesium atoms around it by increasing both the adhesion energy and the equilibrium bond distances. This is at variance with the interaction of metal atoms with the regular MgO ( 100) surface or the Fs defect itself, in which an increase of the adhesion energy is associated with a shortening of the metal-surface distance. The resulting double frustration and cylindrical invariance of the metal-surface interaction cause small gold clusters growing around an Fs nucleation center to be highly fluxional in terms both of rotational freedom and of multiple competing structural motifs. Fragmentation energies of the gold clusters are also discussed, finding that the lowest-energy pathway corresponds to the detachment of a dimer.