We use a Flory-Huggins type approach to describe the swelling of a polymer microgel (a finite size network of polymer chains covalently bound with each other) with immobilized carbon nanotubes in an ionic liquid. The ionic liquid is modeled as a "two-component" solvent consisting of anions and cations. The anions and cations are assumed to have different affinities to the microgel. It is supposed that the carbon nanotubes form a conducting cluster interpenetrating the microgel. We show that if the difference of the Flory-Huggins parameters describing affinities of the anions and the cations to the microgel is high enough, the swelling ratio of the gel can depend on its size.. The physical reason for the phenomenon is the violation of the cations and anions concentrations equality inside the gel that causes a shift of the free energy and can lead to a swelling-collapse transition. The segregation of cations and anions is different for microgels having different sizes because it is opposed by electrostatic field.