We use molecular dynamics (MD) simulations and Polymer Reference Interaction Site Model (PRISM) theory with a coarse-grained (CG) model to study polymer nanocomposites (PNCs) comprised of polymer grafted nanoparticles in a polymer matrix. Specifically, we describe the impact of increasing graft-matrix attraction on the PNC structure quantified in terms of the extent of interpenetration of matrix and graft chains (i.e., grafted layer wetting) and dispersion/aggregation of the grafted particles in the matrix via intermolecular pair correlation functions and structure factors. Past work on PNCs with attractive graft-matrix interactions had already established that grafted layer wetting-dewetting and dispersion-aggregation are two distinct phase transitions with the former being a continuous transition and the latter being a first-order transition with increasing graft-matrix attraction. In this paper, we go beyond that previous work and show that the dispersion and aggregation of polymer grafted particles in PNCs is driven by the hardness and size of the grafted layer which is tuned by the strength of the attractive graft-matrix interactions. As the strength of the graft-matrix attraction increases, graft polymer chains adopt extended conformations to form energetically favorable graft-matrix contacts leading to enhanced grafted layer wetting by matrix chains. This extended grafted layer and increased grafted layer wetting with increasing graft-matrix attraction leads to larger and harder grafted particles compared to analogous PNCs with purely entropic (athermal) graft-matrix interactions. The increased size and hardness of the grafted particles causes the PNC morphology to change from an entropically driven aggregated/dispersed morphology at athermal graft-matrix interaction to a dispersed morphology due to energetically favorable weak graft-matrix attraction, and ultimately, to a correlated fluid of hard grafted particles at strong graft-matrix attraction. We see the above trends in PNCs with isotropic as well as directional attraction between graft and matrix chains, grafted particles at high (densely grafted) and low grafting densities, and with equal graft and matrix chain lengths as well as matrix chain lengths greater than the graft chain lengths.