The topological derivative of cost functionals J that depend on the stress (through the displacement gradient, assuming a linearly elastic material behavior) is considered in a quite general 3D setting where both the background and the inhomogeneity may have arbitrary anisotropic elastic properties. The topological derivative dJ(z) of J quantifies the asymptotic behavior of J under the nucleation in the background elastic medium of a small anisotropic inhomogeneity of characteristic radius a at a specified location z. The fact that the strain perturbation inside an elastic inhomogeneity remains finite for arbitrarily small a makes the small-inhomogeneity asymptotics of stress-based cost functionals quite different than that of the more usual displacement-based functionals. The asymptotic perturbation of J is shown to be of order O(a^3) for a wide class of stress-based cost functionals having smooth densities. The topological derivative of J, i.e. the coefficient of the O(a^3) perturbation, is established, and computational procedures then discussed. The resulting small-inhomogeneity expansion of J is mathematically justified (i.e. its remainder is proved to be of order o(a^3)). Several 2D and 3D numerical examples are presented, in particular demonstrating the proposed formulation of \dJ on cases involving anisotropic elasticity and non-quadratic cost functionals.