We show that primordial black holes - in the observationally allowed mass window with $f_{\rm pbh}=1$ - formed from late nucleating patches in a first order phase transition imply upcoming gravitational wave interferometers will see a large stochastic background arising from the bubble collisions. As an example, we use a classically scale invariant $B-L$ model, in which the right handed neutrinos explain the neutrino masses and leptogenesis, and the dark matter consists of primordial black holes. The conclusion regarding the gravitational waves is, however, expected to hold model independently for black holes coming from such late nucleating patches.