The paradigm of $N$ quantum emitters coupled to a single cavity mode appears in many situations ranging from quantum technologies to polaritonic chemistry. The ideal case of identical emitters is elegantly modeled in terms of symmetric states, and understood in terms of polaritons. In the practically relevant case of an inhomogeneous frequency distribution, this simple picture breaks down and new and surprising features appear. Here we leverage the high degree of control in a strongly coupled cold atom system, where for the first time the ratio between coupling strength and frequency inhomogeneities can be tuned. We directly observe the transition from a disordered regime to a polaritonic one with only two resonances. The latter are much narrower than the frequency distribution, as predicted in the context of ''cavity protection''. We find that the concentration of the photonic weight of the coupled light-matter states is a key parameter for this transition, and demonstrate that a simple parameter based on statistics of transmission count spectra provides a robust experimental proxy for this theoretical quantity. Moreover, we realize a dynamically modulated Tavis-Cumming model to produce a comb of narrow polariton resonances protected from the disorder, with potential applications to quantum networks.