Inelastic electron tunneling spectroscopy (IETS) measurements are usually carried out in the low-voltage ("Ohmic", i.e., linear) regime where the elastic conduction/voltage characteristic is symmetric to voltage inversion. Inelastic features, normally observed in the second derivative d(2)I/dV(2) are also symmetric (in fact antisymmetric) in many cases, but asymmetry is sometimes observed. We show that such asymmetry can occur because of different energy dependences of the two contact self-energies. This may be attributed to differences in contact density of states (different contact material) or different energy dependence of the coupling (STM-like geometry or asymmetric positioning of molecular vibrational modes in the junction). The asymmetry scales with the difference between the energy dependence of these self-energies and disappears when this dependence is the same for the two contacts. Our nonequilibrium Green function approach goes beyond proposed WKB scattering theory(1) in properly accounting for Pauli exclusion, as well as providing a path to generalizations, including consideration of phonon dynamics and higher-order perturbation theory.