A comprehensive study on the analogy of memristive magnetic tunnel junctions and biological neural systems is performed. It is demonstrated that the continuous, bipolar resistance change due to resistive switching can be used to mimic the synaptic connection between neurons. Particularly, the timing-dependent plasticity is observed. Further, the magnetization switching driven by spin-transfer torque in combination with back-hopping is used to generate current spikes. These spikes show remarkable similarities with the signals generated by excited neurons.