We describe a two-scale modeling approach toward anion specificity at surfaces of varying charge and polarity. Explicit-solvent atomistic molecular dynamics simulations at neutral hydrophobic (i.e., nonpolar) and neutral hydrophilic (i.e., polar) self-assembled monolayers furnish potentials of mean force for Na+ and the halide anions F-, Cl-, and I-which are then used within Poisson-Boltzmann theory to calculate ionic distributions at surfaces of arbitrary charge for finite ion concentration. On the basis of calculated lone-ranged electrostatic forces and coagulation properties, we obtain the direct anionic Hofmeister series at negatively charged hydrophobic surfaces. Reversal takes place when going to negative polar or to positive nonpolar surfaces, leading to the indirect series, while for positive polar surfaces the direct series is again obtained. This is in full accordance with a recent experimental classification of colloidal coagulation kinetics and also reflects the trends of the ion specific solubility properties of proteins. A schematic Hofmeister phase diagram is proposed. Partial series reversal is understood as a transient phenomenon for surfaces of intermediate polarity or charge.