Our poster is a presentation of a research project that brings together researchers in artificial intelligence (AI) and conversation analysis (CA): we want to show our first results, but first and foremost we want to address the caveats that we encountered so far in this project and discuss our applied methodology. We conduct a three-year (2021-2024) exploratory study which aims at analyzing task-oriented human-robot interaction (HRI) in order to improve the social/interactional abilities of robots. The focus of this collaborative work is on the multi-modal and situated practices that achieve the temporal coordination of turn-taking (Levinson & Torreira 2015) and the progressivity (Fischer et al. 2019) of interaction, dealing with trouble, repair, reformulations or accounts. In order to focus on these dimensions, we delimited a particular use case and set of actions that will remain the same across different software improvements : the humanoid robot Pepper is supposed to orient and inform users of a university library in Lyon (French language). The poster will show the different steps from (1) the project’s goals, (2) the development of the first version of the robot, taking into account the sequential organization of human-human interactions in service encounters, (3) the data acquisition of 17 hours of video-recorded data (and encountered problems), (4) the sequential and multi-modal analysis of the data with regard to failures and repairs in interaction as for instance the management of the participation framework in multi-party interactions. First collections document how people contingently humanize the robot by indexing preference, rights and obligations in regular interactions, and how they orient to their alignment or not for the practical purpose of building humorous activities or complaints addressed to the other humans. The last two steps are (5) the annotation of approximately 500 interactions (according to the most relevant phenomena identified as problematic or consequential) and (6) the first solutions for a new programming based on developmental machine learning. The corpus has to be systematically annotated so that AI researchers can teach the machine to recognize multimodal features of interaction and propose new AI models that account for sequentiality (prospective and retrospective indexicalities). We address issues with how to mix top-down and bottom-up approaches (Levinson 2013) in order to process indexical and embodied conducts (e.g. humans do not gaze "on their right", they might gaze "at someone" on their right).