The overall mechanical performance of ionic elastomers, such as carboxylated nitrile rubber (XNBR), is largely governed by ionic clusters formed during the cross-linking of the elastomers with zinc oxide. These ionic aggregates promote microphase separation and show additional high-temperature relaxation behavior in dynamic mechanical analysis. In this study, the nature of these ionic aggregates is explored for the first time. We find that some zinc-containing compounds, such as zinc-aluminum-layered double hydroxide and zinc chloride, do not exhibit any extra high-temperature dynamic mechanical relaxation processes, although ionic cross-linking reactions with XNBR occur with all of these zinc compounds. Detailed analysis by Fourier-transform infrared spectroscopy and dynamic mechanical analysis revealed that this high-temperature relaxation behavior does not originate from ionic cross-linking but is associated with the formation of an additional zinc-enriched polymer phase that arises due to reactions between carboxylic groups and zinc oxide. Infrared spectroscopic investigation indicates further that a tetrahedrally coordinated complex facilitates the formation of a zinc-carboxylic polymeric network. Clear microphase separation of the ionic polymer in the elastomer could be directly visualized by transmission electron microscopy for the first time.