Towards a Publicly Available Framework to Process Traceroutes with MetaTrace
Abstract
Various platforms, such as Ark, RIPE Atlas, and M-Lab, conduct millions of traceroutes daily and make them publicly accessible to researchers and operators. While these measurements are easily obtainable, the processing of such data poses challenges. Currently, there is no publicly available general framework designed specifically for storing and querying traceroutes. The conventional method of loading all traceroutes into memory becomes inadequate or necessitates costly hardware as the volume of traceroutes increases significantly. This research paper introduces an innovative utilization of the ClickHouse database, referred to as MetaTrace, and demonstrates its efficiency in querying and analyzing a dataset of 200 million traceroutes enriched with metadata within just 11 seconds, using a single server equipped with reasonably available resources. By making MetaTrace freely accessible as an open-source tool for the community, we aim to establish it as a significant stride toward standardizing traceroute processing. This standardization would enhance the replicability and reproducibility of research outcomes in this domain.
Origin | Files produced by the author(s) |
---|