We developed an iterative scheme to optimize classical mechanical pair-potential parameters of liquids on the basis of diffraction data. An iterative step, similarly as in our previous method, consists of two parts: A classical mechanical simulation with an assumed pair potential and the modification of the potential using the experimental data and quantities calculated in the simulation. The modification part is formulated according to the Gauss-Newton-Marquardt nonlinear parameter fit. The method is used to get tabulated potentials, and seems to be robust also in the case of a fit of few hundred parameters simultaneously. The method keeps the advantages of our previous procedure: The fitted function is the measured structure factor, the Fourier transformation is applied with controllable error, the total functions are determined from the partial ones to avoid the solution of linear equations, and the error of the experiments can be taken into account explicitly. The theoretical justification of the method is the one to one correspondence of the pair potential and the structure factor proved by the inverse theorem of statistical mechanics. We determined effective pair potentials in a tabulated form for liquid mercury at 10 different densities to investigate the structural changes around its metal-nonmetal transition. We found neither a drastic change nor a change in the trends in the structural and pair potential functions. Our results fit well to some of the previous theories about the transition. (C) 2003 American Institute of Physics.