Motivated by the study of linear quadratic optimal control problems, we consider a dynamical system with a constant, quadratic Hamiltonian, and we characterize the number of conjugate times in terms of the spectrum of the Hamiltonian vector field $\vec{H}$. We prove the following dichotomy: the number of conjugate times is identically zero or grows to infinity. The latter case occurs if and only if $\vec{H}$ has at least one Jordan block of odd dimension corresponding to a purely imaginary eigenvalue. As a byproduct, we obtain bounds from below on the number of conjugate times contained in an interval in terms of the spectrum of $\vec{H}$.