Calcium ion binding by the four EF-hand motifs of the protein calmodulin (CaM) is a central event in Ca2+-based cellular signaling. To understand molecular details of this complex process, isolated Ca2+-binding loops can be studied, by use of both experiments and calculations. In this work, we explore the metal specificity of the four Ca2+-binding loops of CaM using density functional theory (DFT) quantum chemical calculations and molecular dynamics simulations. We study CaM complexes with the physiologically important ions of calcium (Ca2+) and magnesium (Mg2+) and also with two other ions, strontium (Sr2+) and lanthanum (La3+). The former is of interest in the area of radioactive waste bioremediation, whereas the latter is often used as a probe of Ca2+-binding sites. We obtain intrinsic metal ion-loop binding energies as well as their components: vacuum, charge-transfer, solvation, entropy, and deformation terms. A detailed analysis of the results reveals that the total binding energy depends on a delicate balance among these energy components. They, in turn, are determined by the cation's charge and size as well as the amino acid composition and flexibility of the loops and the identity of the metal-chelating residues.