Perimeter intrusion detection (PID) deals with the detection of intruders displacing in a protected perimeter. In the video surveillance domain, deep learning has shown tremendous progresses. Existing deep learning based PID systems (PIDS) are supervised and thus require a lot of annotated data. However, since intrusions are rare events, there are very few positives in datasets, thus making them highly imbalanced. Furthermore, a PIDS must adapt to varying real-life scene dynamics, like weather, light, environmental conditions, etc. To address these issues, we propose an autoencoder-based, endto-end trainable, unsupervised PIDS with a module that can adapt to long-term variations in scene dynamics. Our results show competitive performance of the proposed system on the standard i-LIDS dataset.