Equations of the random-phase approximation (RPA) for high-spin open-shell molecules are derived using the time-dependent variational principle and a unitary exponential representation of the high-spin determinantal wave function parametrized with a nonredundant set of variational parameters. The restricted open-shell RPA theory is applied to derive expressions for the polarization propagator, and for the dispersion energy of high-spin open-shell complexes. It is also used to define the intramonomer correlation expansions of the RPA dispersion energy using various Moller-Plesset-type partitionings of the Hamiltonian. A close relation between the present treatment and the multipole expansion approach of Hettema and Wormer [H. Hettema and P. E. S. Wormer, J. Chem. Phys. 93, 3389 (1990)] is established and discussed. Numerical results for various high-spin open-shell-closed-shell and open-shell-open-shell complexes are presented. Comparison of the dispersion energy computed in the random phase approximation with highly correlated results from the full configuration interaction or coupled-cluster singles, doubles, and approximate triples calculations shows that the random phase approximation accounts for the major part of the intramonomer correlation effects in the dispersion energy. For open-shell-closed-shell complexes the convergence of the intramonomer correlation expansion through the second order is very good, while for the more difficult case of open-shell-open-shell complexes is much less satisfactory, and full RPA calculations are necessary to get reliable results. (C) 2003 American Institute of Physics.