In modified theories of gravity, the potentials appearing in the Schrödinger-like equations that describe perturbations of non-rotating black holes are also modified. In this paper we ask: can these modifications be constrained with high-precision gravitational-wave measurements of the black hole's quasinormal mode frequencies? We expand the modifications in a small perturbative parameter regulating the deviation from the general-relativistic potential, and in powers of $M/r$. We compute the quasinormal modes of the modified potential up to quadratic order in the perturbative parameter. Then we use Markov-chain-Monte-Carlo (MCMC) methods to recover the coefficients in the $M/r$ expansion in an ``optimistic'' scenario where we vary them one at a time, and in a ``pessimistic'' scenario where we vary them all simultaneously. In both cases, we find that the bounds on the individual parameters are not robust. Because quasinormal mode frequencies are related to the behavior of the perturbation potential near the light ring, we propose a different strategy. Inspired by Wentzel-Kramers-Brillouin (WKB) theory, we demonstrate that the value of the potential and of its second derivative at the light ring can be robustly constrained. These constraints allow for a more direct comparison between tests based on black hole spectroscopy and observations of black hole `shadows'' by the Event Horizon Telescope and future instruments.