We investigate experimentally and theoretically the multimode dynamics of a two-color quantum dot laser subject to time-delayed optical feedback. We unveil energy exchanges between the longitudinal modes of the excited state triggered by variations of the feedback phase, and observe that the modal competition between longitudinal modes appears independently within the ground state and excited state emission. These features are accurately reproduced with a quantum dot laser model extended to take into account multiple modes for both ground and excited states. Finally, we discuss the significant impact of such behavior on feedback-based control of two-color quantum dot lasers.