Ultrasound Localization Microscopy (ULM) is an ultrasound imaging technique that relies on the acoustic response of sub-wavelength ultrasound scatterers to map the microcirculation with an order of magnitude increase in resolution as compared to conventional ultrasound imaging. Initially demonstrated in vitro, ULM has matured and sees implementation in vivo for vascular imaging of organs or tumors in both animal models and humans. The performance of localization algorithms greatly defines the quality of vascular mapping. Here, we compiled and implemented a collection of ultrasound localization algorithms and devised three in silico and in vivo datasets to compare their performance through 11 metrics. We also present two novel algorithms designed to increase speed and overall performance. By providing a comprehensive open package to perform ULM that includes localization algorithms, the datasets used, and the evaluation metrics, we aim to equip researchers with a tool to identify the optimal localization algorithm for their