Niobium (Nb)-doped diamond-like carbon (Nb-DLC) films were deposited on Si and stainless steel substrates using a hybrid deposition system composed of a high power impulse magnetron sputtering (HiPIMS) and a pulse direct current magnetron sputtering (PDC-MS). The Nb concentration (0-23.43 at. %) in the film was controlled by adjusting the PDC power. The influences of Nb doping on the composition, phase structure, microstructure, and carbon atoms bond of DLC films were studied from various measurement tools, such as X-ray photoelectron spectroscope, X-ray diffraction, transmission electron microscope, scanning electron microscope, atomic force microscope and Raman spectroscope. Nano-indentation, stress-tester and ball-on-disk tribometer were also applied to study the mechanical properties and friction behavior, respectively. The results indicated that with increasing Nb concentration, the Nb-DLC films transformed from amorphous phase to nanocomposite structure with carbide phase embedding in the matrix of amorphous carbon. The surface morphology was composed of large agglomeration hills and the cross-sectional images exhibited apparent columnar structure. In addition, the increase in Nb content led to an increase of hardness initially, and then decreased with further increase of Nb content. The residual stress was decreased monotonically. The pure DLC film exhibited a lower friction coefficient and narrower width of wear scar, while the nanocomposite Nb-DLC films showed a high friction coefficient and large wear scar, although they possessed a high hardness. It could be ascribed to the formation of carbide hard particles as a result of scratching the contact surface during friction tests at high Nb doping concentration.