Axial variations in geometry and presence of viscous displaced fluid are known to alter the diffusive-dynamics of capillary imbibition of a wetting liquid. We here show that the coupled effect of axially varying capillary geometry and finite viscosity of the displaced fluid can lead to significant variations in both short and long time dynamics of imbibition. Based on a theoretical model and lattice Boltzmann simulations, we analyze capillary displacement of a viscous liquid in straight and diverging capillaries. At short times, the imbibition length scales proportionally with time as opposed to the diffusive-dynamics of imbibition of a single wetting liquid. Whereas, at long times, geometry-dependent power-law behavior occurs which qualitatively resembles single liquid imbibition. The distance at which the crossover between these two regimes occurs depends strongly on the viscosities of the imbibing and the displaced liquid. Additionally, our simulations show that the early time imbibition dynamics are also affected by the dynamic contact angle of the meniscus.