Filamentary structures extraction in medical and biological images is a challenging problem. Muscular/Neural fibers, neurites and blood arteries are some examples. Their delineation is particularly problematic due to the lack of solid visual support that is also compromised by the presence of clutter and low signal to noise ratios. In this article, we propose a modular approach to curvilinear structures extraction based on recent advances in discrete optimization on the basis of aggregate clustering. Given an initial clustering of the detected points, first a pair-wise Markov Random Field (MRF) is considered to determine consistent elongated structures while penalizing their number and rejecting outliers. The outcome of this process is then locally refined through a non-submodular MRF aiming at center-line extraction process and guided by local geometric consistency between segments expressing the intra-cluster variability. Promising results for microtubules delineation in TIRFM images as well as for guidewires segmentation in fluoroscopic images demonstrate the potentials of the method.