With the aid of ab initio calculations, an n-body potential of the Ni-Nb system is constructed under the Finnis-Sinclair formalism and the constructed potential is capable of not only reproducing some static physical properties but also revealing the atomistic mechanism of crystal-to-amorphous transition and associated kinetics. With application of the constructed potential, molecular dynamics simulations using the solid solution models reveal that the physical origin of crystal-to-amorphous transition is the crystalline lattice collapsing while the solute atoms are exceeding the critical solid solubilities, which are determined to be 19 atom % Ni and 13 atom % Nb for the Nb- and Ni-based solid solutions, respectively. It follows that an intrinsic glass-forming ability of the Ni-Nb system is within 19-87 atom % Ni, which matches well with that observed in ion beam mixing/solid-state reaction experiments. Simulations using the Nb/Ni/Nb (Ni/Nb/Ni) sandwich models indicate that the amorphous layer at the interfaces grows in a layer-by-layer mode and that, upon dissolving solute atoms, the Ni lattice approaches and exceeds its critical solid solubility faster than the Nb lattice, revealing an asymmetric behavior in growth kinetics. Moreover, an energy diagram is obtained by computing the energetic sequence of the NixNb100-x alloy in fcc, bcc, and amorphous structures, respectively, over the entire composition range, and the diagram could serve as a guide for predicting the metastable alloy formation in the Ni-Nb system.