Matrix spectral factorization is traditionally described as finding spectral factors having a fixed analytic pole configuration. The classification of spectral factors then involves studying the solutions of a certain algebraic Riccati equation, which parametrizes their zero structure. The pole structure of the spectral factors can also be parametrized in terms of solutions of another Riccati equation. We study these two Riccati equations and describe how they can be combined for the construction of general spectral factors, which involve both zero and pole flipping on an arbitrary reference spectral factor.