Using neutron reflectivity, the internal structure of polyelectrolyte multilayers is described on the nanoscale. Each film consists of a protonated and a deuterated block, built from x protonated and y deuterated polycation/polyanion bilayers, respectively. The number of bilayers N = x + y is kept constant; the position of the interface between the blocks is varied systematically. The polyanion is poly(styrenesulfonate) (PSS), and the polycation is poly(allylamine hydrochloride) (PAH) or poly(diallyldimethylammonium chloride) (PDADMAC). Always, the first four to five bilayers are thinner than the average bilayer thickness, but the three terminating bilayers are sometimes thicker. In the core zone, the bilayer thickness is constant. The internal roughness is smallest next to the film/air interface and increases with the number of bilayers away from the film/air interface. This suggests that each deposition step promotes the interdiffusion of the supporting layers. At the selected preparation conditions, the internal roughness increases more for PDADMAC/PSS than for PAH/PSS; the diffusion constants differ by 2 orders of magnitude.