Engagement is the process by which participants establish, maintain, and end their perceived connection. Automatic engagement inference is one of the tasks required to develop successful human-centered HMI applications. Engagement is a multi-faceted multimodal construct requiring high accuracy in interpretating contextual, verbal and non-verbal cues, making the development of an intelligent automated engagement inference system challenging. Existing surveys concentrate on specific application settings, and a comprehensive survey covering the different engagement facets, definition and inference across various contexts is lacking. Moreover, despite the importance of context-aware modeling, the literature lacks a systematic context-aware overview on the topic. This paper presents a comprehensive survey on previous work in engagement for HMI, entailing interdisciplinary definition, engagement components, publicly available datasets, ground truth assessment, and commonly used features and methods, serving as a guide for the development of future HMI interfaces with reliable context-aware engagement inference capability. An in-depth review across embodied and disembodied interaction modes, and an emphasis on the interaction context of which engagement is studied sets apart this survey from existing ones. Our findings suggest four important directions for future research: (1) context-aware computational modeling, (2) temporal dynamics, (3) personalised computing, and (4) bias and fairness of engagement inference systems.