Online Clustering of Massive Text Data Streams
Résumé
Clustering large text data streams is a complex and computationally demanding task due to its unique challenges including high dimensionality, data sparsity, evolving data distributions, and the curse of dimensionality. Traditional clustering techniques often struggle to keep up with these dynamic changes and high-velocity nature of streaming text data while maintaining efficiency and accuracy. In this paper, we introduce OMTStream, a novel approach that combines an adaptive text representation technique with an online clustering based CluStream. Our method continuously learns meaningful semantic representations of text while reducing dimensionality, allowing for fast and efficient real-time clustering. By leveraging micro-clusters that evolve over time, OMTStream can track emerging patterns and adapt to shifting data distributions without the need for retraining from scratch. We evaluate our approach on real-world datasets, demonstrating its superior performance compared to existing techniques. Through a comprehensive set of experiments, we also validate the effectiveness of OMTStream across various scenarios, highlighting its robustness, adaptability, and computational efficiency in handling large, continuously evolving text data streams.
Domaines
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| Licence |