k-mer matrix compression
Compression de matrices de k-mers
Résumé
The analysis of k-mers matrices, essential in agronomy, health and ecology, poses significant challenges regarding Big Data storage and processing. Recently, the tool kmtricks [1] was proposed for creating such matrices and was later used in the kmindex tool [2], designed for indexing large datasets. This opened the doors to indexing terabytes or even petabytes of genomic data. However, one of the main limitations resides in the size of the created matrices, in the order of several terabytes.
In this work, we propose a block compression method on top of the kmtricks matrices, when used by kmindex for indexing and query purposes. The main objective is to achieve sensible compression ratios with a limited impact on the index creation time and on the query time. This method enables partial and targeted decompression, thus optimizing data processing. The BlockCompressor plugin, integrated to kmindex, slices matrices into fixed-size blocks, compressed individually using the LZMA algorithm. The positions of the blocks are encoded using the Elias-Fano succinct data structure [3,4]. This allows both storage of a large number of positions and efficient random accesses.
To improve the compressibility of matrices, we also exploit an idea inspired by phylogenetic compression [5]. Hence, the impact of reordering matrix columns based on phylogenetic order is studied and used as a means of increasing their compressibility. Preliminary results show that this reordering significantly improves matrix compressibility.
Finally, the approach we propose reduces the storage space required, with a limited impact on query time, making k-mer matrices more accessible and usable.
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| Licence |