We report on the salt-induced aggregation of citrate-coated gold nanoparticles (AuNPs) confined within poly(N-isopropylacrylamide) (PNIPAM) brushes grafted from flat substrates. Compared to highly dispersed AuNPs, a red-shift and broadening of the surface plasmon (SP) band is observed when the AuNPs are confined by the PNIPAM brush matrix due to their close-vicinity. Additional red-shifting and broadening occur upon immersion in aqueous salt solutions (1 M NaF, NaCl, NaBr, and KCl). Nanoparticle assemblies are established due to salt-induced aggregation of AuNPs and are dependent on the type of salt. In the presence of KCl, nanoparticle assemblies are built up that result in the formation a second plasmon peak at similar to 700 nm. The color change of PNIPAM/AuNP is associated with (1) the collapse of the PNIPAM brushes in the presence of salt and (2) nanoparticle aggregation due to electrostatic screening of the negative charges around the AuNPs by the salt ions. Ion specificity is related to ion-pair association energies and adsorption behavior of ions at the AuNP surface. In addition, we perform a neutron reflectivity experiment to resolve the internal structure of swollen PNIPAM/AuNP hybrids and find that penetrated AuNPs cause PNIPAM chain stretching due to electrostatic repulsion between charged particles in the brush.