The effects of realistic roughness and elasticity on the interactions between charged silica spheres are studied as a function of surface potential, screening length, interfacial energy, and roughness. The repulsive force F-rep that must be overcome to bring charged spheres into contact is relatively insensitive to elasticity unless spheres are hundreds of times softer than silica. F-rep is also insensitive to roughness and interfacial energy. In contrast, roughness has a large effect on the binding energy of spheres and the force F-sep to separate them. Both are lowered by 1 to 2 orders of magnitude by the measured surface roughness of less than 1 nm on 1 mu m silica spheres. The reason is that interactions between rigid spheres are dominated by the highest surface peaks rather than the entire spherical surface. Elasticity can increase the pull-off force of rough spheres by a factor of 2 or more because additional surface area can be brought into contact. The implications of these results for shear-thickening transitions are discussed.