This paper is concerned with the billiard version of Jacobi's last geometric statement and its generalizations. Given a non-focal point $O$ inside an elliptic billiard table, one considers the family of rays emanating from $O$ and the caustic $\Gamma_n$ of the reflected family after $n$ reflections off the ellipse, for each positive integer $n$. It is known that $\Gamma_n$ has at least four cusps and it has been conjectured that it has exactly four (ordinary) cusps. The present paper presents a proof of this conjecture in the special case when the ellipse is a circle. In the case of an arbitrary ellipse, we give an explicit description of the location of four of the cusps of $\Gamma_n$, though we do not prove that these are the only cusps.