The main advantage of the Boundary Element Method (BEM) is that only the domain boundaries are discretized. It is thus well-suited to study site effects. This advantage is offset by the full BEM matrix. In the last couple of years, fast BEMs have been proposed to overcome this drawback. If the efficiency of fast BEMs has been demonstrated, the iteration count becomes now the main limitation to consider realistic problems. Mesh adaptation is an additional technique to reduce the computational cost and number of iterations of the BEM. In this contribution, we give an overview of recent works to speed-up fast BEMs, i.e. an anisotropic metric-based mesh adaptation technic.