We present a trainable hierarchical architecture capable of detecting newness (or outliers) at all hierarchical levels. This contribution paves the way for deep neural architectures that are able to learn in an incremental fashion, for which the ability to detect newness is an indispensable prerequisite. We verify the ability to detect newness by conducting experiments on the MNIST database, where we introduce either localized changes, by adding noise to a small patch of the input, or global changes, by changing the global arrangement of local patterns which is not detectable at the local level.