We extend umbrella sampling with replica exchange steps to calculate free energies that are important in the self-assembly of peptides. This leads to a more than 10-fold speed up over conventional umbrella sampling, thereby providing a practical method to calculate these free-energy differences. This approach can also observe first-order phase transitions and pinpoint the location of the concomitant boundary. When conformational changes are involved, this method can handle peptides up to similar to 7 residues, providing a rapid and accurate assessment of the thermodynamic properties of model systems, and can thus be used to answer fundamental questions about peptide self-assembly. When no major conformational changes are involved, we expect the size limit to be equal to that of standard molecular dynamics.