Raman scattering has been performed on single crystal hydrogenated silicon and germanium surfaces to investigate the temperature dependence of their vibrational spectral profiles. Based on a single-mode dephasing model for pure vibrational dephasing, the frequency shift and line broadening were analyzed to extract the following dephasing parameters: exchange mode frequency, coupling strength, and friction parameter. The exchange modes for the XH stretches on hydrogenated X(100) surfaces (X=Ge, Si, and C) are found to match their respective bending frequencies. The corresponding ones for hydrogenated X(111) surfaces, on the other hand, are located within the bulk phonon. This surface dependence of the exchange mode in surface vibrational dephasing dynamics is correlated with the structural relaxation and its associated symmetry variation at surfaces. It is further confirmed by the experiments performed on deuterated semiconductor surfaces. A site-symmetry induced representation method is exploited to analyze the symmetry properties of the surface vibrations on these two surfaces. This group-theoretical analysis has revealed the selection rules of choosing the dominant exchange modes on hydrogenated semiconductor surfaces. We have identified the exchange modes in the surface phonon spectra of these surfaces according to their symmetry properties.