The modification of vibrational dynamics is essential for controlling chemical reactions and IR photonic applications. The hybridization between cavity modes and molecular vibrational modes provides a new way to control molecular dynamics. In this work, we study the dynamics of molecular vibrational polaritons in various solvent environments. We find the dynamics of the polariton system is strongly influenced by the nature of the solvents. While the relaxation from upper polariton (UP) to dark modes is always fast (<5 ps) regardless of the medium, lower polariton (LP) in low polarity solvents shows much slower transfer (10-30 ps) into dark modes, despite the fact that the LP lifetime remains within 5 ps. This result suggests that in the latter media, the energy pumped into the LP is first transferred into intermediate states which only subsequently decay into dark modes. In contrast, in solvent environments that strongly interact with the solute, the LP population relaxes into the dense dark state manifold within a much faster time scale. We propose the intermediate state to be the high-lying excited states of dark modes, which are effectively populated by LP via, e.g., ladder-climbing. Such population in the high-lying states can be retained for tens of picoseconds, which could be pertinent to recently observed cavity-modified chemistry.