Computation of viscoplastic fluid flows has always been a challenging task. Viscoplastic models are intrinsically discontinuous at the yielded-unyielded interface, which leads to numerical difficulties, because of the singularity in the Jacobian matrix of the resulting discretized equations. For this reason, several modeling or numerical approaches have been proposed, the most popular being the Papanastasiou regularization (PR) and the Augmented Lagrangian (AL) methods, respectively. Recently, studies on AL methods have focused on developing accelerated algorithms, since the required computational cost of using AL is extremely high. In the present work, a fast converging and efficient algorithm is proposed for tracking the yield surface and predicting the flow field of viscoplastic fluids accurately. The numerical procedure is the Penalized Augmented Lagrangian (PAL) method, which is based on a monolithic Newton solver for AL, where the governing equations of the Lagrange-multiplier tensor for both the rate-of-strain projection and the extra-stress tensors are penalized. To test the efficiency of our algorithm, five benchmark flow-problems with fixed, free and moving boundaries are studied. First, the problem of the steady rise of a bubble in a viscoplastic medium is addressed validating the new algorithm with the findings by Dimakopoulos et al. (2013). Then the entrance flow in a rectangular channel is solved, where a primary unyielded region is found around the centerline in the developed part of the flow and secondary unyielded regions near the entrance. In addition, the lid-driven cavity problem is solved, which is an often used test for various numerical algorithms and the results are compared to relevant studies for viscoplastic fluids such as those of Syrakos et al. (2013, 2014) and Treskatis et al. (2016). Furthermore, the developed flow in a square duct is examined, similarly to Saramito (2016). Finally, the transient filament stretching of a shearthinning, yield stress fluid is examined, and the results are compared to those by Balmforth et al. (2010). In all cases, either steady or transient, the algorithm captures the yield surfaces correctly, while maintaining a low computational cost, because the convergence of the method requires only a few (i.e. 5-30) Newton iterations. Based on these extensive tests, PAL is found to be superior combining accuracy and speed to all existing solution methods for viscoplastic fluids.