Given their tendency to be incorporated into the core during differentiation, the highly-siderophile elements (HSEs) in Earth's mantle are thought to have been accreted as a 'late veneer' after the end of the giant impact phase. Bottke et al (2010) proposed that the large Earth-to-Moon HSE abundance ratio can be explained if the late veneer was characterized by large (D = 1000-4000km) impactors. Here we simulate the evolution of the terrestrial planets during a stochastic late veneer phase from the end of accretion until the start of the late heavy bombardment ~500 Myr later. We show that a late veneer population of 0.05 Earth masses dominated by large (D > 1000km) bodies naturally delivers a ~0.01 Earth mass veneer to Earth, consistent with constraints. The eccentricities and inclinations of the terrestrial planets are excited by close encounters with the largest late veneer bodies. We find the best agreement with their post-veneer orbits if either a) the terrestrial planets' pre-veneer angular momentum deficit AMD_0 was less than half of the current one AMD_now, or b) AMD_0 <= AMD_now and the veneer was limited to smaller (D_max <= 2000km) bodies. Veneer impacts on Venus, Earth and Mars were mostly accretionary but on Mercury and the Moon they were mostly erosive. In ~20% of simulations an energetic impact occurred that could have removed >25% of Mercury's mass, thereby increasing its iron mass fraction. We show that, due to the erosive nature of larger impacts, the Moon cannot accrete any material from objects larger than 500-1000km. The large Earth-to-Moon HSE abundance ratio is explained if the late veneer included large impactors (D >= 500-1000km) regardless of their size distribution, as long as most of Earth's veneer came from large bodies. The spin angular momenta imparted by stochastic late veneer impacts was far in excess of the current ones for Mercury and Venus.