Following inflation, the Universe may pass through an early matter-dominated phase supported by the oscillating inflaton condensate. Initially small fluctuations in the condensate grow gravitationally on subhorizon scales and can collapse to form nonlinear ``inflaton halos''. Their formation and subsequent tidal interactions will source gravitational waves, resulting in a stochastic background in the present Universe. We extend N-body simulations that model the growth and interaction of collapsed structures to compute the resulting gravitational wave emission. The spectrum of this radiation is well-matched by semi-analytical estimates based on the collapse of inflaton halos and their tidal evolution. We use this semi-analytic formalism to infer the spectrum for scenarios where the early matter-dominated phase gives way to a thermalized universe at temperatures as low as $100\,\mathrm{MeV}$ and we discuss the possible experimental opportunities created by this signal in inflationary models in which thermalization takes place long after inflation has completed.