We consider nonlinear scalar-input differential control systems in the vicinity of an equilibrium. When the linearized system at the equilibrium is controllable, the nonlinear system is smoothly small-time locally controllable, i.e., whatever $m>0$ and $T>0$, the state can reach a whole neighborhood of the equilibrium at time $T$ with controls arbitrary small in $C^m$-norm. When the linearized system is not controllable, we prove that small-time local controllability cannot be recovered from the quadratic expansion and that the following quadratic alternative holds.Either the state is constrained to live within a smooth strict invariant manifold, up to a cubic residual, or the quadratic order adds a signed drift in the evolution with respect to this manifold. In the second case, the quadratic drift holds along an explicit Lie bracket of length $(2k+1)$, it is quantified in terms of an $H^{-k}$-norm of the control, it holds for controls small in $W^{2k,\infty}$-norm. These spaces are optimal for general nonlinear systems and are slightly improved in the particular case of control-affine systems. Unlike other works based on Lie-series formalism, our proof is based on an explicit computation of the quadratic terms by means of appropriate transformations. In particular, it does not require that the vector fields defining the dynamic are smooth. We prove that $C^3$ regularity is sufficient for our alternative to hold.This work underlines the importance of the norm used in the smallness assumption on the control: depending on this choice of functional setting, the same system may or may not be small-time locally controllable, even though the state lives within a finite dimensional space.