A novel displacement principle of metal nanoparticles for target analysis, differing from the usual target-induced aggregation principle, was proved feasible by the use of para-aminothiophenol coupled Au nanoparticles (PATP-Au) multilayer as probes to detect Hg2+. The PATP-Au multilayer was fabricated through layer-by-layer assembly of Au nanoparticles on optical waveguide (OWG) chip surface using para-aminothiophenol (PATP) as coupling molecules. The localized surface plasmon resonance (LSPR) extinction from Au nanoparticles and the PATP as a Raman reporter enable to easily capture changes in PATP-Au multilayer by OWG and of surface enhanced Raman spectroscopy, respectively. The introduction of the Hg2+, which has a higher binding affinity to the thiol group of PATP, greatly destroyed the multilayer structure, and produced a large change, several folds higher than the noise, in LSPR features and Raman signals of PATP-Au multilayer probes, and resulted in an excellent selectivity for Hg2+ detection at a low level of 1 nM. This investigation provides us more ideas on the future development of surface analysis techniques for the detection of various target analytes. (C) 2012 Elsevier Inc. All rights reserved.