A time resolved infrared-infrared double resonance technique is used to study the vibrational relaxation of O-3 in rare gas and nitrogen matrices. A tunable infrared (IR) pulsed source excites the nu(1) + nu(3) level of O-3 in the ground electronic state. A continuous wave (cw) CO2 laser probes the populations of the fundamental and upsilon(2)=1 levels as a function of time. After minimization of thermal effects, the relaxation signal can be analyzed. At fixed probe frequency, the behavior of the rise time of the signals with the pump frequency shows spectral diffusion to occur inside the inhomogeneous profiles. At high concentration in argon (O-3/Ar=1/250), intermolecular energy transfer is observed between the two sites. In xenon matrices, it has time to take place at concentrations 1/2000. The relaxation rates of the upsilon(2)=1 level to the ground state are measured at different concentrations in rare gas and nitrogen matrices. At high dilution, a maximum relaxation time, called intrinsic relaxation time tau(i), is determined in the different matrices: it covers three orders of magnitude, from a few hundred nanoseconds in neon to 320 microseconds in xenon. The results are discussed and compared with literature data within the frame of the isolated binary collision model.