We study experimentally a sediment of self-propelled Brownian particles with densities ranging from dilute to ergodic supercooled, to nonergodic glass, to nonergodic polycrystal. In a compagnon letter, we observe a nonmonotonic response to activity of relaxation of the nonergodic glass state: a dramatic slowdown when particles become weakly self-propelled, followed by a speedup at higher activities. Here we map ergodic supercooled states to standard passive glassy physics, provided a monotonic shift of the glass packing fraction and the replacement of the ambient temperature by the effective temperature. However we show that this mapping fails beyond glass transition. This failure is responsible for the nonmonotonic response. Furthermore, we generalize our finding by examining the dynamical response of an other class of nonergodic systems : polycrystals. We observe the same nonmonotinic response to activity. To explain this phenomenon, we measure the size of domains were particles move in the same direction. This size also shows a nonmonotonic response, with small lengths corresponding to slow relaxation. This suggests that the failure of the mapping of nonergodic active states to a passive situation is general and is linked to anisotropic relaxation mechanisms specific to active matter.