Glass optics with ultra-low roughness surfaces (<2 angstrom rms) are strongly desired for high-end optical applications (e.g., lasers, spectroscopy, etc.). The complex microscopic interactions that occur between slurry particles and the glass workpiece during optical polishing ultimately determine the removal rate and resulting surface roughness of the workpiece. In this study, a comprehensive set of 100 mm diameter glass samples (fused silica, phosphate, and borosilicate) were polished using various slurry particle size distributions (PSD), slurry concentrations, and pad treatments. The removal rate and surface roughness of the glasses were characterized using white light interferometry and atomic force microscopy. The material removal mechanism for a given slurry particle is proposed to occur via nano-plastic deformation (plastic removal) or via chemical reaction (molecular removal) depending on the slurry particle load on the glass surface. Using an expanded Hertzian contact model, called the Ensemble Hertzian Multi-gap (EHMG) model, a platform has been developed to understand the microscopic interface interactions and to predict trends of the removal rate and surface roughness for a variety of polishing parameters. The EHMG model is based on multiple Hertzian contacts of slurry particles at the workpiece-pad interface in which the pad deflection and the effective interface gap at each pad asperity height are determined. Using this, the interface contact area and each particle's penetration, load, and contact zone are determined which are used to calculate the material removal rate and simulate the surface roughness. Each of the key polishing variables investigated is shown to affect the material removal rate, whose changes are dominated by very different microscopic interactions. Slurry PSD impacts the load per particle distribution and the fraction of particles removing material by plastic removal. The slurry concentration impacts the areal number density of particles and fraction of load on particles versus pad. The pad topography impacts the fraction of pad area making contact with the workpiece. The glass composition predominantly impacts the depth of plastic removal. Also, the results show that the dominant factor controlling surface roughness is the slurry PSD followed by the glass material's removal function and the pad topography. The model compares well with the experimental data over a variety of polishing conditions for both removal rate and roughness and can be extended to provide insights and strategies to develop practical, economic processes for obtaining ultra-low roughness surfaces while simultaneously maintaining high material removal rates.