The solid-state structure of mercury oxide and its low-pressure modifications are known to significantly differ from those found for the corresponding zinc and cadmium compounds, that is, one changes from a simple hexagonal wurtzite or cubic rock salt structure found in zinc oxide and cadmium oxide to unusual chainlike montroydite and cinnabar structures in mercury oxide. Here, we present relativistic and nonrelativistic density functional studies which demonstrate that this marked structural difference is caused by relativistic effects. For HgO, the simple rock salt structure is only accessible at higher pressures. Relativistic effects reduce the cohesive energy by 2.2 eV per HgO unit. and decrease the density of the crystal by 14% due to a change in the crystal symmetry. Band structure and density of states calculations also reveal large changes in the electronic structure due to relativistic effects, and we argue that the unusual yellow to red color of HgO is a relativistic effect as well.