The recently proposed coupled channel density matrix (CCDM) method for nondissipative dynamics [L. Pesce and P. Saalfrank, Chem. Phys. 219, 43 (1997)], is extended to open quantum systems. This method, which is the density matrix analogue of the coupled channel wave packet (CCWP) method in Schrodinger wave mechanics, allows for the solution of nuclear Liouville-von Neumann equations in more than one dimension including unbound modes. A semiphenomenological, Markovian, and trace-conserving dissipative model within the dynamical semigroup approach is suggested, and efficient numerical schemes for its implementation are presented. Using a two-mode model, we apply the dissipative CCDM method to the problem of vibrationally excited gas-phase hydrogen molecules, relaxing during the scattering from a cold, metallic, and nondissociative surface. The significance of a relaxation mechanism based on electron-hole pair creation in a metallic substrate is addressed. The dependence of the survival probability of the vibrationally excited molecules on the dissipative model parameters, on their initial translational energy, and on isotopic substitution is examined and rationalized on the basis of a simple classical kinetic model.