Peptide self-assembly plays a role in a number of diseases, in pharmaceutical degradation, and in emerging biomaterials. Here, we aim to develop an accurate molecular-scale picture of this process using a multiscale computational approach. Recently, Shell (Shell, M. S. J. Chem. Phys. 2008, 129, 144108-7) developed a coarse-graining methodology that is based on a thermodynamic quantity called the relative entropy, a measure of how different two molecular ensembles behave. By minimizing the relative entropy between a coarsegrained peptide system and a reference all atom system, with respect to the coarse grained model's force field parameters, an optimized coarse-grained model can be obtained. We have reformulated this methodology using a trajectory-reweighting and perturbation strategy that enables complex coarse-grained models with at least hundreds of parameters to be optimized efficiently. This new algorithm allows for complex peptide systems to be coarse grained into much simpler models that nonetheless recapitulate many correct features of detailed all-atom ones. In particular, we present results for a polyalanine case study, with attention to both individual peptide folding and large-scale fibril assembly.