Amorphous polymers in their glass transition regime can be described as a tiling of nanometric domains. Each domain exhibits its own relaxation time, which is distributed over at least four decades. These domains are known as dynamical heterogeneities. This article describes the mechanics of amorphous polymers using a stochastic continuum mechanics model that includes their heterogeneous dynamics. Solving this model both by finite elements and by using a self-consistent method, we find a viscoelastic relaxation spectrum quantitatively similar to an experimentally measured spectrum in a polymer. We show evidence that elastic couplings between domains control the stress relaxation after a step strain and result in a narrowing of the long-time region of the viscoelastic spectrum (as compared to that of dynamical heterogeneities).