This work proposes a new unsupervised deep generative model for system logs. It is designed to be generic and may be used in various downstream anomaly detection tasks, such as system failure or intrusion detection. It is based on the (reasonable) assumption that most log lines follow rather fixed syntactic structures, which enables us to replace the costly traditional convolutional and recurrent architectures by a much faster component: a deep averaging network. Our model still exploits a standard recurrent model with attention to capture the dependencies between successive log lines. We experimentally validate the proposed generative model on a real dataset obtained from a state-of-the-art High Performance Computing cluster and show the effectiveness of the proposed approach in modeling the "normal" behaviour of the system.