By using theoretical analysis and molecular dynamics simulations, we investigate the structure of colloidal crystals formed by nonmagnetic microparticles (or magnetic holes) suspended in ferrofluids (called inverse ferrofluids), by taking into account the effect of polydispersity in size of the nonmagnetic microparticles. Such polydispersity often exists in real situations. We obtain an analytical expression for the interaction energy of monodisperse, bidisperse, and polydisperse inverse ferrofluids. Body-centered tetragonal (bct) lattices are shown to possess the lowest energy when compared with other sorts of lattices and thus serve as the ground state of the systems. Also, the effect of microparticle size distributions (namely, polydispersity in size) plays an important role in the formation of various kinds of structural configurations. Thus, it seems possible to fabricate colloidal crystals by choosing appropriate polydispersity in size.