The purpose of this article is to show that when dynamically cold, dissipationless self-gravitating systems collapse, their evolution is a strong function of the symmetry in the initial distribution. We explore with a set of pressure-less homogeneous fluids the time-evolution of ellipsoidal distributions and map the depth of potential achieved during relaxation as function of initial ellipsoid axis ratios. We then perform a series of $N$-body numerical simulations and contrast their evolution with the fluid solutions. We verify an analytic relation between collapse factor ${\\cal C}$ and particle number $N$ in spherical symmetry, such that ${\\cal C} \\propto N^{1/3}$. We sought a similar relation for axisymmetric configurations, and found an empirical scaling relation such that ${\\cal C} \\propto N^{1/6}$ in these cases. We then show that when mass distributions do not respect spherical- or axial-symmetry, the ensuing gravitational collapse deepens with increasing particle number $N$ but only slowly: 86% of triaxial configurations may collapse by a factor of no more than 40 as $N\\to\\infty$. For $N\\approx 10^5$ and larger, violent relaxation develops fully under the Lin-Mestel-Shu instability such that numerical $N$-body solutions now resolve the different initial morphologies adequately.