Using pump-probe technique, the dynamics of the hot carriers in metallic nanodots induced by femtosecond laser pulses are investigated in gold nanoparticles embedded in hydrogel and in organic gel and compared to that in aqueous solution. We found that changing the surrounding matrix from aqueous solution to hydrogel and then to organic gel leads to a large increase in the relaxation time of both the electron-phonon (e-ph) and the phonon-phonon (ph-ph) coupling. Furthermore, the ph-ph relaxation time becomes sensitive to the type of the organic solvent trapped in the gel network. This indicates that the relaxation dynamics depend on the thermal conductivity, chemical structure and the molecular dynamics of the surrounding medium.