Data flow management has become a key network activity, strengthening the need for efficient data flow classification tools. However, pervasive encryption of communication has dramatically jeopardised the legacy tools. Recent advances in Deep Learning offer a wide variety of architectures that seem relevant for this purpose. These architectures are based on different data representations as input of their classification process. In this paper, we show the need for a deeper understanding of the features used by Deep Learning models to perform such classification. Our objective is to exploit this knowledge for defining a better data processing so that the chosen architecture will significantly improve the classification process. We will show that some information carried by packet headers need to be analyzed through a separate process. This analysis highlight that current Deep Learning approaches in the literature fail to classify encrypted flows in practice. We therefore propose a new modular Deep Learning architecture called Servername Protocol Packet Network (SPPNet) to overcome this drawback. We will show by a proof of concept that SPPNet allows to perform real-time network flow classification at packet level.