Classical dictionary learning methods simply normalize dictionary columns at each iteration, and the impact of this basic form of regularization on generalization performance (e.g. compression ratio on new images) is unclear. Here, we derive a tractable performance measure for dictionaries in compressed sensing based on the low $M^*$ bound and use it to regularize dictionary learning problems. We detail numerical experiments on both compression and inpainting problems and show that this more principled regularization approach consistently improves reconstruction performance on new images.