This article addresses one of the components of our ongoing work towards an efficient computational modeling methodology for evaluating all effects on a submerged structure of a remote underwater explosion. Following up on a previous study devoted to computing the transient acoustic fields induced by the shock wave initially sent by the blast on a rigid submarine, we focus here on the second stage of the underwater event, namely solving the transient fluid-structure interaction (FSI) between the structure and the incompressible potential flow induced by the delayed, and slower, oscillations of the gas bubble created by the remote blast. The boundary element method (BEM) is the best-suited approach for handling potential flow problems in large fluid domains (idealized as unbounded), whereas the finite element method (FEM) naturally applies to the transient structure analyses. To perform the FEM-BEM coupling we use a sub-cycling approach that alternates fluid and solid analyses with Neumann boundary conditions. The transient nature of the coupled analysis and the recourse to sub-cycling together make the overall procedure rely on a large number of BEM potential flow solutions, while the complexities of the wet surface and of the solid transient response imply a need for large BE models for the flow potential. This combination of reasons mandates accelerating the BE component. Accordingly, our main contribution is to study the feasibility and effectiveness of coupling the Hierarchical-matrix accelerated BEM (H-BEM) and the FEM for the FSI problems of interest. In particular, we show that the same integral operators can be used at all time instants in spite of the expected global motion of the submerged structure, a feature that the H-BEM can exploit to full advantage. The proposed original treatment is validated against analytical solutions for the case of a motionless or mobile rigid spherical immersed object, and then tested on a complex configuration representative of target applications.