The hydrodynamics of bubble columns is revisited with a focus on the increase of the apparent relative velocity observed in the heterogeneous regime. We report experiments in an air-water bubble column of diameter 0.4m using a new Doppler probe that gives access to bubble velocities conditioned by the void fraction. First, meso-scale structures corresponding to high (clusters), low (voids) and intermediate gas concentrations relative to the mean gas hold-up are unambiguously identified using Voronoï tessellations built from the phase indicator function. Second, conditional bubble velocity data demonstrate that meso-scale structures drive the transport of both phases in the heterogeneous regime. Finally, the unconditional relative velocity is recovered from conditional relative velocities weighted by the probability of presence of bubbles in a given structure. These results provide a physical origin for swarm factors introduced in simulations. They also offer a way to connect the relative velocity enhancement with meso-scale structures characteristics, and to advance our understanding of scaling rules.