Magnetic anisotropy is a key component in the design of single-molecule magnets (SMMs) possessing a large barrier height for magnetization reversal Lanthanide based SMMs are the most promising candidates in this arena as they offer a large magnetic anisotropy due to the presence of strong spin orbit coupling. Among lanthanides, Er(III) complexes are gaining attention in the area of SMMs, because of their intriguing magnetic properties and attractive blocking temperatures Here, we have undertaken detailed ab initio calculations on four structurally diverse Er(III) SMMs to shed light on how the magnetic anisotropy is influenced by the role of symmetry and structural distortions The employed CASSCF+RASSI calculations have offered rationale for the observed differences in the estimated U-eff values for the studied complexes and also offered hints to the mechanism of magnetic relaxation. The differences in the mechanism of magnetic relaxations are further analyzed based on the Er-ligand interactions, which is obtained by analyzing the charges, densities, luminescent behavior and the frontier molecular orbitals. Our calculations, for the first time, have highlighted the importance of high symmetry environment and ligand donor strength in obtaining large U-eff values for the Er(III) complexes. We have examined these possibilities by modeling several structures with variable coordination numbers and point group symmetry. These results signify the need of a detailed understanding on the shape of the anisotropy and the point group symmetry in order to achieve large U-eff values in Er(III) single ion magnets.