RESPEC is a new framework that introduces residue specificity into elastic network modeling (ENM) to successfully render intact protein-ligand complexes as well as apo proteins. This framework establishes a broader application of coarse-graining idea via describing (i) a coarse-grained residue/node through its heavy atoms as virtual nodes, (ii) an effective B-factor for such a node, directly obtained from the experimental data, and (iii) a node-node interaction by a cumulative distance-dependent force constant. RESPEC improves the level of correlations with B-factors after optimizing the parameters of the model. In the absence of ligands, the mean correlations exceed 0.72, which is higher than the classical ENM results, based on a diverse set of proteins. Global modes satisfactorily describe the conformational transitions for apo structures. When the ligands are included at atomistic resolution in RESPEC calculations, mean correlation values exceed 0.9 over the same data set.