The physical properties of metals change when their dimensions are reduced to the nano-scale and new phenomena like the Localized Surface-Plasmon Resonance (LSPR) appear. This collective electronic excitation can be tuned over a large spectral range by adapting the material, size and shape. The existing literature is as rich as controversial as e.g. size-dependent spectral shifts of the LSPR in small metal nanoparticles, induced by quantum effects, are reported to the red, to the blue or entirely absent. Here we report how complementary experiments on mass-selected small silver nanoparticles embedded in silica can yield inconsistent results on the same system: while optical absorption shows no size-effect in the range between only a few atoms and ~10 nm, a clear spectral shift is observed in single-particle electron spectroscopy. Our quantitative interpretation, based on a mixed classical/quantum model, resolves the apparent contradictions, not only within our experimental data, but also in the literature. Our comprehensive model describes how the local environment is the crucial parameter controlling the manifestation or absence of size effects.