Replica exchange molecular dynamics is used to study the effect of different rigid-body (mTIP3P, TIP4P, SPC/E) and hybrid (H1.56, H3.00) water models on the conformational free energy landscape of the alanine oligopeptides (acAnme and acA(5)nme), in conjunction with the CHARMM22 force field. The free energy landscape is mapped out as a function of the Ramachandran angles. In addition, various secondary structure metrics, solvation shell properties, and the number of peptide solvent hydrogen bonds are monitored. Alanine dipeptide is found to have similar free energy landscapes in different solvent models, an insensitivity which may be due to the absence of possibilities for forming i-(i + 4) or i-(i + 3) intrapeptide hydrogen bonds. The pentapeptide, acA(5)nme, where there are three intrapeptide backbone hydrogen bonds, shows a conformational free energy landscape with a much greater degree of sensitivity to the choice of solvent model, though the three rigid-body water models differ only quantitatively. The pentapeptide prefers nonhelical, non-native PPII and beta-sheet populations as the solvent is changed from SPC/E to the less tetrahedral liquid (H1.56) to an LJ-like liquid (H3.00). The pentapeptide conformational order metrics indicate a preference for open, solvent-exposed, non-native structures in hybrid solvent models at all temperatures of study. The possible correlations between the properties of solvent models and secondary structure preferences of alanine oligopeptides are discussed, and the competition between intrapeptide, peptide solvent, and solvent solvent hydrogen bonding is shown to be crucial in the relative free energies of different conformers.