Seismometers on board the Apollo and InSight missions measured seismic wave velocities in lunar and Martian surface material (i.e., regolith). As on Earth, these regoliths have different grain size distributions. To better interpret these in-situ seismic observations we conducted laboratory experiments to investigate the role of grain size heterogeneity on seismic wave velocity. Our experiments use piezoelectric bender elements to excite and measure seismic wave propagation through samples comprised of binary mixes of glass beads. We propose a physical explanation of the experimental observations, which cannot be attributed to bulk density variations alone. To isolate the effect of grain size distribution, 14 binary mixes were intentionally designed with different grain size distributions, and our results indicate that two mechanisms influence the sample microstructure depending on grain size distribution: hole filling and contact rupture. While the former leads to increases in seismic velocity, the latter does the opposite. Consequently, particle size heterogeneity should be considered when investigating seismic wave velocities in planetary surfaces.