Expert-in-the-Loop Supervised Learning for Computer Security Detection Systems
Apprentissage supervisé et systèmes de détection : une approche de bout-en-bout impliquant les experts en sécurité
Abstract
Supervised detection models can be deployed in detection systems, as an adjunct to traditional
detection techniques, to strengthen detection. Supervised learning has been successfully applied to
various computer security detection problems: Android applications, PDF, or portable executable
files to mention only the most obvious examples. Despite these encouraging results, there remain
some significant barriers to the widespread deployment of machine learning in operational detection
systems.
The standard supervised learning pipeline consists of data annotation, feature extraction, training
and evaluation. Security experts must carry out all these steps to set up supervised detection
models ready for deployment. In this thesis, we adopt an end-to-end approach. We work on
the whole machine learning pipeline with security experts as its core since it is crucial to pursue
real-world impact.
First of all, security experts may have little knowledge about machine learning. They may
therefore have difficulty taking full advantage of this data analysis technique in their detection
systems.
This thesis provides methodological guidance to help security experts build supervised detection
models that suit their operational constraints. Moreover, we design and implement DIADEM, an
interactive visualization tool that helps security experts apply the methodology set out. DIADEM
deals with the machine learning machinery to let security experts focus mainly on detection.
Besides, most research works assume that a representative annotated dataset is available for
training while such datasets are particularly expensive to build in computer security. Active learning
has been introduced to reduce expert effort in annotation projects. However, it usually focuses on
minimizing only the number of manual annotations, while security experts would rather minimize
the overall time spent annotating. Moreover, user experience is often overlooked while active
learning is an interactive procedure that should ensure a good expert-model interaction.
This thesis proposes a solution to effectively reduce the labeling cost in computer security
annotation projects. We design and implement an end-to-end active learning system, ILAB, tailored
to security experts needs. Our user experiments on a real-world annotation project demonstrate
that security experts can gather an annotated dataset with a low workload thanks to ILAB.
Finally, feature extraction is usually implemented manually for each data type. Nonetheless,
detection systems process many data types and designing a feature extraction method for each
of them is tedious. Automatic feature generation would significantly ease, and thus foster, the
deployment of machine learning in detection systems.
In this thesis, we define the constraints that such methods should meet to be effective in building
detection models. We compare three state-of-the-art methods based on these criteria, and we point
out some avenues of research to better tailor these techniques to computer security experts needs.
L’objectif de cette thèse est de faciliter l’utilisation de l’apprentissage supervisé dans les systèmes de détection pour renforcer la détection. Dans ce but, nous considérons toute la chaîne de traitement de l’apprentissage supervisé (annotation, extraction d’attributs, apprentissage, et
évaluation) en impliquant les experts en sécurité.
Tout d’abord, nous donnons des conseils méthodologiques pour les aider à construire des modèles de détection supervisés qui répondent à leurs contraintes opérationnelles. De plus, nous concevons et nous implémentons DIADEM, un outil de visualisation interactif qui aide les experts en sécurité à appliquer la méthodologie présentée. DIADEM s’occupe des rouages de l’apprentissage supervisé pour laisser les experts en sécurité se concentrer principalement sur la détection.
Par ailleurs, nous proposons une solution pour réduire le coût des projets d’annotations en sécurité informatique. Nous concevons et implémentons un système d’apprentissage actif complet, ILAB, adapté aux besoins des experts en sécurité. Nos expériences utilisateur montrent qu’ils peuvent annoter un jeu de données avec une charge de travail réduite grâce à ILAB.
Enfin, nous considérons la génération automatique d’attributs pour faciliter l’utilisation de l’apprentissage supervisé dans les systèmes de détection. Nous définissons les contraintes que de telles méthodes doivent remplir pour être utilisées dans le cadre de la détection de menaces.
Nous comparons trois méthodes de l’état de l’art en suivant ces critères, et nous mettons en avant des pistes de recherche pour mieux adapter ces techniques aux besoins des experts en sécurité.
Origin | Files produced by the author(s) |
---|
Loading...