The aggregation kinetics of thermoresponsive core-shell micelles with a poly(N-isopropyl acrylamide) shell in pure water or in mixtures of water with the cosolvents methanol or ethanol at mole fractions of 5% is investigated during a temperature jump across the respective cloud point. Characteristically, these mixtures give rise to cononsolvency behavior. At the cloud point, aggregates are formed, and their growth is followed with time-resolved small-angle neutron scattering. Using the reversible association model, the interaction potential between the aggregates is determined from their growth rate in dependence on the cosolvents. The effect of the cosolvent is attributed to the interaction potential on the structured layer of hydration water around the aggregates. It is surmised that the latter is perturbed by the cosolvent and thus the residual repulsive hydration force between the aggregates is reduced. The larger the molar volume of the cosolvent, the more pronounced is the effect. This framework provides a molecular-level understanding of solvent-mediated effective interactions in polymer solutions and new opportunities for the rational control of self-assembly in complex soft matter systems.