We present a novel method to learn human preferences during, and for, the execution of concurrent joint human-robot tasks. We consider tasks realized by a team of a human operator and a robot helper that should adapt to the human's task execution preferences. Different human operators can have different abilities, experiences, and personal preferences, so that a particular allocation of activities in the team is preferred over another. We cast the behavior of concurrent multi-agent cooperation as a semi Markov Decision Process and show how to model and learn human preferences over the team behavior. After proposing two different interactive learning algorithms, we evaluate them and show that the system can effectively learn and adapt to human preferences.