Memory is the fundamental form of temporal complexity: when present but uncontrollable, it manifests as non-Markovian noise; conversely, if controllable, memory can be a powerful resource for information processing. Memory effects arise from/are transmitted via interactions between a system and its environment; as such, they can be either classical or quantum in nature. From a practical standpoint, quantum processes with classical memory promise near-term applicability: they are more powerful than their memoryless counterpart, yet at the same time can be controlled over significant timeframes without being spoiled by decoherence. However, despite practical and foundational value, apart from simple two-time scenarios, the distinction between quantum and classical memory remains unexplored. We first analyse various physically-motivated candidates regarding a suitable definition for classical memory that lead to remarkably distinct phenomena in the multi-time setting. Subsequently, we systematically characterise the hierarchy of multi-time memory effects in quantum mechanics, many levels of which collapse in the two-time setting, thereby making our results genuinely multi-time phenomena.