The phenomenon of metachronal waves in cilia carpets has been well known for decades; these waves are widespread in biology, and have fundamental physiological importance. While it is accepted that in many cases cilia are mainly coupled together by the hydrodynamic velocity field, a clear understanding of which aspects determine the collective wave properties is lacking. It is a difficult problem, because both the behavior of the individual cilia and their coupling together are nonlinear. In this work, we coarsegrain the degrees of freedom of each cilium into a minimal description in terms of a configuration-based phase oscillator. Driving colloidal particles with optical tweezers, we then experimentally investigate the coupling through hydrodynamics in systems of many oscillators, showing that a collective dynamics emerges. This work generalizes to a wider class of systems our recent finding that the non-equilibrium steady state can be understood based on the equilibrium properties of the system, i.e. the positions and orientations of the active oscillators. In this model system, it is possible to design configurations of oscillators with the desired collective dynamics. The other face of this problem is