Band 3 is an anion exchanger for chloride/bicarbonate in the plasma membrane of erythrocytes. The function of Band 3 may be influenced by the interactions between Band 3 and lipids or lipid domains in the plasma membrane. In this work, using atomistic molecular dynamics simulation, we investigate the interactions between Band 3 and nanosized lipid domains in ternary lipid bilayers composed of saturated lipids, unsaturated lipids, and cholesterol. The simulations show asymmetric interactions between Band 3 and lipid nanodomains in the two leaflets of a neutral lipid bilayer with lower cholesterol concentration. With an increase in cholesterol concentration in the bilayer, cholesterol affects the interactions between Band 3 and lipid domains by deforming the structure of the protein. Additionally, the anionic lipids, which prefer to bind to some specific sites of Band 3, also affect the interactions between Band 3 and lipid domains. This work provides some new insight into understanding the distribution of Band 3 in the plasma membrane of erythrocytes as well as its anion exchange function.