The spectral element method is analyzed for the simulation of viscoelastic fluid flows in plane and cylindrical geometries. The implementation of the method and choices of functional spaces is discussed in detail. The method is then applied to the case of the Maxwell-B fluid and compared with the so-called 4 x 4 SUPG finite element method on the perturbed channel flow problem. Analysis of the rate of convergence of the method is provided as well as thorough discussion on the decomposition of the geometry in spectral elements: the choice is between the h-vs. p-refinement. A comparison with other numerical (e.g. pseudo-spectral) methods is performed for the flow of the Maxwell-B fluid in a wavy tube. In each test case, very good agreement is found.