Low-temperature specific heat and inelastic light scattering experiments have been performed on a series of cesium borate glasses and on a cesium borate crystal. Raman measurements on the crystalline sample have revealed the existence of cesium rattling modes in the same frequency region where glasses exhibit the boson peak (BP). These localized modes are supposed to overlap with the BP in cesium borate glasses affecting its magnitude. Their influence on the low frequency vibrational dynamics in glassy samples has been considered, and their contribution to the specific heat has been estimated. Evidence for a relation between the changes of the BP induced by the increased amount of metallic oxide and the variations of the elastic medium has been provided.