Orbits tracked by ellipsoids immersed in inviscid and viscous environments are studied by means of Kirchhoff's equations and high resolution numerical simulations using a variant of the immersed boundary method. We explore the consequences of Kozlov & Onishchenko (1982)'s theorem of non-integrability of Kirchhoff's equations to show how the fraction of phase space in chaotic orbits is sensitively determined by the body shape, fluid/ solid density ratio and the fraction of initial energy in rotational motion. We show how the added mass tensor of the system is an important player in both viscous and inviscid flow, in causing chaos in a triaxial ellipsoid while acting to suppress it in a spheroid. We identify a new integral of motion for a spheroid in inviscid fluid: one component of the generalised angular momentum. A spheroid, which can never execute chaotic dynamics in inviscid flow, is shown to display chaos in viscous flow due to irregular vortex shedding. But the dynamics of the spheroid is restricted whether in viscous or in inviscid flow, unlike in the triaxial ellipsoid, due to our extra integral of motion.