Numeracy—the ability to understand and reason with numbers—is essential for interpreting and acting on quantitative information, yet it remains under-addressed in visualization research. In this position paper, we argue that numeracy should be a guiding principle in the design and evaluation of public-facing visualizations. Drawing on cognitive psychology and decision science, we propose a numeracy-informed design process: (1) measuring individual differences in numeracy, (2) designing visualizations that support gist and verbatim understanding, reduce cognitive effort, and provide evaluative meaning, and (3) evaluating quantitative comprehension across numeracy levels. Our goal is to spark a broader conversation about inclusive visualization practices that reflect numerical diversity and support public understanding of quantitative information.