O'Grady studied recently $m$-th modified diagonals for a smooth projective variety, generalizing the Gross-Schoen modified small diagonal. These cycles $\Gamma^m(X,a)$ depend on a choice of reference point $a\in X$ (or more generally a degree $1$ zero-cycle). We prove that for any $X,a$, the cycle $\Gamma^m(X,a)$ vanishes for large $m$. We also prove the following conjecture of O'Grady: if $X$ is a double cover of $Y$ and $\Gamma^m(Y,a)$ vanishes (where $a$ belongs to the branch locus), then $\Gamma^{2m-1}(X,a)$ vanishes, and we provide a generalization to higher degree finite covers. We finally prove the vanishing $\Gamma^{n+1}(X,o_X)=0$ when $X=S^{[m]}$, $S$ a $K3$ surface, and $n=2m$, which was conjectured by O'Grady and proved by him for $m=2,3$.