Building on previous work for 3D inverse scattering in the frequency domain, this article develops the concept of topological derivative for 3D elastic and acoustic-wave imaging of media of arbitrary geometry using data in the time domain. The topological derivative, which quantifies the sensitivity of the cost functional associated with the inverse scattering problem due to the creation at a specified location of an infinitesimal hole (for the elastodynamic case) or rigid inclusion (for the acoustic case), is found to be expressed in terms of the time convolution of the free field and a supplementary adjoint field. The derivation of the topological derivative follows the generic pattern proposed in previous studies, which is transposable to a variety of other physical problems. A numerical example, where the featured cost function is defined in terms of synthetic data arising from the scattering of plane acoustic waves by a rigid spherical inclusion, illustrates the utility of the topological derivative concept for defect identification using time-varying data.