Building k-nn graphs from large text data - Archive ouverte HAL Access content directly
Conference Papers Year : 2014

Building k-nn graphs from large text data

Pietro Michiardi
  • Function : Author
  • PersonId : 1084771
Olivier Thonnard
  • Function : Author
  • PersonId : 965107
Wim Mees
  • Function : Author
  • PersonId : 773581
  • IdRef : 152672567


In this paper we present our new design of NNCTPH, a scalable algorithm to build an approximate k-NN graph from large text datasets. The algorithm uses a modified version of Context Triggered Piecewise Hashing to bin the input data into buckets, and uses NN-Descent, a versatile graph-building algorithm , inside each bucket. We use datasets consisting of the subject of spam emails to experimentally test the influence of the different parameters of the algorithm on the number of computed similarities, on processing time, and on the quality of the final graph. We also compare the algorithm with a sequential and a MapRe-duce implementation of NN-Descent. For our datasets, the algorithm proved to be up to ten times faster than NN-Descent, for the same quality of produced graph. Moreover, the speedup increased with the size of the dataset, making NNCTPH a sensible choice for very large text datasets.
Fichier principal
Vignette du fichier
bigdata2014_debatty.pdf (82.87 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01133324 , version 1 (19-03-2015)



Thibault Debatty, Pietro Michiardi, Olivier Thonnard, Wim Mees. Building k-nn graphs from large text data. IEEE International Conference on Big Data (Big Data), 2014, Oct 2014, Washington DC, United States. ⟨10.1109/BigData.2014.7004276⟩. ⟨hal-01133324⟩


52 View
201 Download



Gmail Facebook Twitter LinkedIn More