We consider periodic orbits of controlled hybrid dynamic systems and want to find open-loop controls that yield maximally stable limit cycles. Instead of optimizing the spectral or pseudo-spectral radius of the monodromy matrix A, which are non-smooth criteria, we propose a new approach based on the smoothed spectral radius rho(alpha)(A), a differentiable criterion favorable for numerical optimization. Like the pseudo-spectral radius, the smoothed spectral radius rho(alpha)(A) converges from above to the exact spectral radius rho(A) for alpha -> 0. Its derivatives can be computed efficiently via relaxed Lyapunov equations. We show that our new smooth stability optimization program based on rho(alpha)(A) has a favorable structure: it leads to a differentiable nonlinear optimal control problem with periodicity and matrix constraints, for which tailored boundary value problem methods are available. We demonstrate the numerical viability of our method using the example of a walking robot model with nonlinear dynamics and ground impacts as a complex open-loop stability optimization example.