The physiological mechanisms of the neural coding of colors aim at explaining how physical colors, i.e. reflected or emitted light entering the eyes from a visual environment, are converted into perceived colors, i.e. the colors that can be sensed by humans. These mechanisms are well-known to involve both three separate receptor types, the LMS cones, and spectrally opponent and non-opponent interactions resulting from the activity rates of ganglion and lateral geniculate nucleus cells. Color perception is thus a process inherently linked to an observing apparatus and to an experimental environment. The main objective of this contribution is to present a rigorous mathematical model that allows, by taking into account both trichromacy and color opponency viewpoints, to explain relativistic color perception phenomena first argued by Yilmaz in 1962. The cornerstone of the proposed approach is the description of Hering's opponency by means of a quantum system, namely a rebit, whose (chromatic) states are shown to satisfy Einstein-Poincaré relativistic addition law. This implies for instance that the Hilbert metric on the state space of the rebit is relevant to express a chromatic constancy property with respect to observer changes, as confirmed by measurements on experimental data.