This paper considers the controllability problem for both continuous- and discrete-time linear heterogeneous multi-agent systems with directed and weighted communication topology. First, two kinds of neighbour-based control protocols based on the distributed protocol of first-order and second-order multi-agent systems are proposed, under which it is shown that a heterogeneous multi-agent system is controllable if the underlying communication topology is controllable. Then, under special leader selection, the result shows that the controllability of a heterogeneous multi-agent system is solely decided by its communication topology graph. Furthermore, some necessary and/or sufficient conditions are derived for controllability of communication topology from algebraic and graphical perspectives. Finally, simulation examples are presented to demonstrate the effectiveness of the theoretical results.