The concept of topological derivative has proved effective as a qualitative inversion tool for a wave-based identification of finite-sized objects. Although for the most part, this approach remains based on a heuristic interpretation of the topological derivative, a first attempt toward its mathematical justification was done in Bellis et al. (Inverse Problems 29:075012, 2013) for the case of isotropic media with far field data and inhomogeneous refraction index. Our paper extends the analysis there to the case of anisotropic scatterers and background with near field data. Topological derivative-based imaging functional is analyzed using a suitable factorization of the near fields, which became achievable thanks to a new volume integral formulation recently obtained in Bonnet (J. Integral Equ. Appl. 29:271-295, 2017). Our results include justification of sign heuristics for the topological derivative in the isotropic case with jump in the main operator and for some cases of anisotropic media, as well as verifying its decaying property in the isotropic case with near field spherical measurements configuration situated far enough from the probing region.