The relationship between the self-affine structure of cold-deposited films and the surface-enhanced Raman (SERS) intensity of benzene adsorbed on the films is examined. Based on variable temperature STM studies the structure of cold-deposited silver films is shown to be self-affine with a fractal dimension similar to2.6, more or less independent of temperature for T less than similar to 270 K. The fractal structure is shown to collapse to a more or less compact structure when the films are annealed to similar to 280 K. SERS activity ceases at a somewhat lower temperature (similar to 250 K) for all the films examined. SERS enhancements rise by factors as great as 8 as a function of annealing temperature over and above their values at the lowest deposition temperatures used (24 K). The rise reaches a maximum at a temperature that depends both on the deposition temperature of the films and the excitation wavelength. (Such an annealing effect on the SERS intensity has been known for many years.) We suggest that the observations are consistent with recently developed theories of the optical properties of fractal clusters so long as one includes arguments regarding the optimal dimensions of the monomers comprising those clusters whose mean value change during annealing. (Fractality, however, is not an essential condition for observing intense SERS.) The disappearance of the SERS signal before the self-affine structure of the film collapses is, therefore, attributed to the monomer size having significantly passed its optimal size before the temperature at which the collapse of the fractal nature of the film is reached. We argue that this collapse is mediated by surface diffusion. The narrow temperature range over which this occurs is intriguing and worthy of a proper theoretical investigation.