Absorber layers of semiconductor saturable absorber mirrors are required to show high absorption modulation with recovery times on the order of 100 fs and low nonsaturable losses. While to provide such fast recovery times, defect states for carrier trapping need to be incorporated into the absorber material, defects and surface roughness can cause additional losses in nonsaturable absorption. A specially designed GaAs/fluoride multilayer stack was grown to study the relation between growth conditions, surface roughness, nonsaturable losses, and absorption modulation. The growth of the multilayer stack included three epitaxial growth regimes: (a) homoepitaxial growth of GaAs on GaAs (111) B, (b) heteroepitaxy of CaF2 on GaAs, and (c) heteroepitaxy of GaAs on CaF2. While the homoepitaxial and first CaF2 layer growth proceeded two dimensional, island nucleation was obtained for the GaAs absorber and top CaF2 layer. The CaF2 surface was exposed to an electron beam of different doses to increase free surface energy for subsequent GaAs overgrowth and the surface roughness of the absorber layer was found to decrease with increasing electron dose. Nonlinear properties and light scattering were measured and con-elated to the growth parameters. Linear reflectivity and absorption modulation were close to the theoretical values of the designed multilayer stack fur the region exposed to the highest electron dose. With a recovery time of about 500 fs, the grown GaAs absorber layer is an excellent choice for an all-optical switching application in broadband AlCaAs/CaF2 semiconductor saturable absorber mirrors.