The phonon Boltzmann transport equation (BTE) describes micro- and nanoscale heat transfer, incorporating phonon dispersion and polarization. Its numerical solution, especially in the nongray case, is still a challenging problem. In this work, we have adapted the semi-Lagrangian method, originally developed for fluid dynamics, to the nongray phonon BTE. Although the developed method is explicit, the time step can be chosen independent from the Courant-Friedrichs-Lewy limit. Merely from accuracy consideration, the time step is limited by the relaxation time. The stability and accuracy are numerically verified with analytical results in one dimension and numerical results in the literature in two dimensions, in both gray and nongray settings and from diffusive to ballistic heat transfer regimes. (C) 2019 Elsevier Ltd. All rights reserved.