During evaporative drying of a colloidal dispersion, the structural behavior at the air-dispersion interface is of particular relevance to the understanding of the consolidation mechanism and the final structural and mechanical properties of the porous media. The drying interface constitutes the region of initial drying stress that, when accumulated over a critical thickness, leads to crack formation. This work presents an experimental study of top down drying of colloidal silica dispersions with three different sizes (radius 5, 8, and 13 nm). Using specular neutron reflectivity, we focus on the structural evolution at the free drying front of the dispersion with a macroscopic drying surface and demonstrate the existence of a thick concentrated surface layer induced by heterogeneous evaporation. The reflectivity profile contains a strong structure peak due to scattering from particles in the interfacial region, from which the interparticle distance is deduced. A notable advantage of these measurements is the direct extraction of the corresponding dispersion concentration from the critical total reflection edge, providing a straightforward access to a structure-concentration relation during the drying. process. The bulk reservoir of this experimental configuration renders it possible to verify the evaporation-diffusion balance to construct the surface layer and also to check reversibility of particle ordering. We follow the structural evolution of this surface layer from a sol to a soft wet-gel that is the precursor of a fragile skin and the onset of significant particle aggregation that precedes formation of the wet-crust. Separate complementary measurements on the structural evolution in the bulk dispersion are also carried out by small-angle neutron scattering, where the particle concentration is also extracted directly from the experimental curves. The two sets of data reveal similar structural evolution with concentration at the interface and in the bulk and an increase in the degree of ordering with the particle size.