The autonomous learning of different objects in images, with a continual and unsupervised context, relies on detecting unknown objects and recognizing known ones based on the learned visual representation. Novelty detection is challenging because of the internal representation drifts of known objects not been seen for a long time. Most existing approaches either perform offline unsupervised learning on a large dataset, or continual supervised learning. Nevertheless, very few existing approaches propose unsupervised continual learning for object recognition. In this paper, we propose a new neural network-based approach for continually learning representations of objects from image sequences, that is able to autonomously detect novel objects and to recognize previously learned ones during training. It is based on a statistical test, performed on internal representations, adapted to counterbalance the concept drift, without storing any image. Experimental results show that our approach outperforms the state of the art on MNIST and Fashion-MNIST datasets. In particular, our approach avoids to over-segment the distribution of clusters, which artificially increases traditional indicators such as clustering accuracy.