This paper focuses on the structure and the concept of the framework used in the vesselness filters benchmark that was recently introduced. Vesselness filters are used to detect the presence of vessels in an image. There exists a wide variety of such filters and comparing their respective strengths and weaknesses is a non-trivial task, especially given the different contexts in which they are published. This benchmark was designed to ease such comparison process whereas remaining easy to customize. More specifically, this paper presents the benchmark structure and architecture. It also shows how to integrate new vesselness filters and/or new metrics in the benchmark with the requirements for future comparisons and online demonstrations.