The energy spectrum of a hole in the valence bands of a Xe crystal which is bound to a negative point charge, is calculated and analyzed in form of a correlation diagram relating the individual bands of the valence band structure to the level scheme of the bound hole states. It is found that this energy spectrum depends critically on the whole band structure, and in particular on the splitting of the j = 3/2 bands. Our calculated binding energies are in good agreement with recent experimental results.