Among amorphous states, glass is defined by relaxation times longer than the observation time. This nonergodic nature makes the understanding of glassy systems an involved topic, with complex aging effects or responses to further out-of-equilibrium external drivings. In this respect active glasses made of self-propelled particles have recently emerged as stimulating systems which broadens and challenges our current understanding of glasses by considering novel internal out-of-equilibrium degrees of freedom. In previous experimental studies we have shown that in the ergodicity broken phase, the dynamics of dense passive particles first slows down as particles are made slightly active, before speeding up at larger activity. Here, we show that this nonmonotonic behavior also emerges in simulations of soft active Brownian particles and explore its cause. We refute that the Deadlock by Emergence of Active Directionality (DEAD) model we proposed earlier describes our data. However, we demonstrate that the nonmonotonic response is due to activity enhanced aging, and thus confirm the link with ergodicity breaking. Beyond self-propelled systems, our results suggest that aging in active glasses is not fully understood.