Citrate-reduced silver colloids (CRSCs) are used extensively for surface-enhanced Raman scattering (SERS) studies of cations but are typically found to be ineffective for detection of anions unless they are treated with compounds that give them positively charged coatings. In this work CRSCs which were suitable for detection of anions were generated by treatment with aggregating agents that did not bind strongly to the silver surface. Under these conditions the major factor determining the enhancement of added anions was their ability to displace whatever anions were already present. In the case of CRSCs, residual citrate was observed when the colloids were aggregated with sulfate salts, since neither sulfate nor the residual nitrate displaced it. On addition of more strongly binding anions, such as halides, the citrate was displaced and the bands of the added analyte appeared, allowing them to be detected without the need for creation of positively charged coatings. It was found that the relative affinities of the anions, as determined by displacement experiments monitored by SERS, followed the solubilities of their silver salts, presumably because both properties are strongly dependent on the strength of the Ag-anion bonds. The relative affinities determine which anions can be detected in the presence of which others; nitrate, sulfate, and perchlorate are lower in the series than citrate and so are not observed. Displacement experiments show that dipicolinic acid (DPA) and Cl- have similar (stronger) binding, but they can be displaced in turn by Br- and I-, which have the highest affinity and lowest solubility. This model allows a broad range of previous observations to be rationalized and allows the experimental conditions suitable for detection of particular new analytes to be designed on rational principles.