We investigate the segregation of impurities to grain boundaries in colloidal polycrystalline monolayers using video microscopy. A model colloidal alloy is prepared by embedding large spherical impurities in a polycrystalline monolayer of small host colloidal hard spheres, which stops grain growth at a finite grain size. The size ratio between the impurities and the host particles determines whether they behave as interstitial or substitutional impurities in the bulk crystal, akin to those in real alloys. We find that the partitioning of impurities between the grains and the grain boundaries is in very good agreement with the Langmuir McLean adsorption model for equilibrium grain boundary segregation. This enables the direct measurement of the free energy of adsorption for the two types of impurities. Near saturation, we characterize the spatial distribution of the adsorbed impurities and find that it strongly depends on their interstitial or substitutional nature. This is because the relative importance of clustering and mixing due to nonadditivity is determined by geometrical constraints imposed by the crystalline host lattice.