3D rapid transient acoustic problems are difficult to solve numerically when dealing with large geometries, because numerical methods based on geometry discretisation (mesh), such as the boundary element method (BEM) or the finite element method (FEM), often require to solve a linear system (from the spacial discretisation) for each time step. We propose a numerical method to efficiently deal with 3D rapid transient acoustic problems set in large exterior domains. Using the Z-transform and the convolution quadrature method (CQM), we first present a straightforward way to reframe the problem to the solving of a large amount (the number of time steps, M) of frequency-domain BEMs. Then, taking advantage of a well-designed high-frequency approximation (HFA), we drastically reduce the number of frequency-domain BEMs to be solved, with little loss of accuracy. The complexity of the resulting numerical procedure turns out to be O(1) in regards to the time discretisation and O(N log N) for the spacial discretisation, the latter being prescribed by the complexity of the used fast BEM solver. Examples of applications are proposed to illustrate the efficiency of the procedure in the case of fluid-structure interaction: the radiation of an acoustic wave into a fluid by a deformable structure with prescribed velocity, and the scattering of an abrupt wave by simple and realistic geometries.