Paramagnetic molecules can show long spin-coherence times, which make them good candidates as quantum bits (qubits). Reducing the efficiency of the spin-phonon interaction is the primary challenge toward achieving long coherence times over a wide temperature range in soft molecular lattices. The lack of a microscopic understanding about the role of vibrations in spin relaxation strongly undermines the possibility of chemically designing better-performing molecular qubits. Here we report a first-principles characterization of the main mechanism contributing to the spin-phonon coupling for a class of vanadium(IV) molecular qubits. Post-Hartree-Fock and density functional theory methods are used to determine the effect of both intermolecular and intramolecular vibrations on modulation of the Zeeman energy for four molecules showing different coordination geometries and ligands. This comparative study provides the first insight into the role played by coordination geometry and ligand-field strength in determining the spin- lattice relaxation time of molecular qubits, opening an avenue to the rational design of new compounds.