Specific ion effects, related to the hydration of ions and ion-solute interactions, play a fundamental part in many processes in chemistry and biology. Although intensively studied since the seminal studies of Franz Hofmeister and co-workers, their molecular origin has only recently started to be unveiled. In this work, we have investigated the interaction between halide anions and a selected set of amino acid residues in an attempt to identify the forces behind ion specificity. Two-dimensional potential energy surfaces have been calculated with the use of local second order Moller-Plesset perturbation theory (LMP2), coupled with the COSMO model to describe solvent effects. The results show in great detail the impact of dispersion interactions, in particular for the heavier anions (Br- and I-). The obtained potential energy surfaces also hint at a greater mobility of iodide in the vicinity of a residue, which correlates well with its placing in the Hofmeister series.