Interaction is a dynamic process that evolves in real time. Participants interpret and orient themselves towards turns of speech based on expectations of relevance and social/conversational norms (that have been extensively studied in the field of Conversation analysis). A true challenge to Human Robot Interaction (HRI) is to develop a system capable of understanding and adapting to the changing context, where the meaning of a turn is construed based on the turns that have come before. In this work, we identify issues arising from the inadequate handling of the sequential flow within a corpus of in-the-wild HRIs in an open-world university library setting. The insights gained from this analysis can be used to guide the design of better systems capable of handling complex situations. We finish by surveying efforts to mitigate the identified problems from a natural language processing/machine dialogue management perspective.