We show that the elliptic genus of the higher rank E-strings can be computed based solely on the genus 0 Gromov-Witten invariants of the corresponding elliptic geometry. To set up our computation, we study the structure of the topological string free energy on elliptically fibered Calabi-Yau manifolds both in the unrefined and the refined case, determining the maximal amount of the modular structure of the partition function that can be salvaged. In the case of fibrations exhibiting only isolated fibral curves, we show that the principal parts of the topological string partition function at given base-wrapping can be computed from the knowledge of the genus 0 Gromov-Witten invariants at this base-wrapping, and the partition function at lower base-wrappings. For the class of geometries leading to the higher rank E-strings, this leads to the result stated in the opening sentence.