Compositional relations represent a good source of information in the task of scene understanding. However, current approaches in domestic service robotics only scratch the surface of the benefits of compositional relations by leveraging only their spatial component. In this position paper, we propose a new perspective on the use of compositional relations as a means to extract meaning from context in open-ended interactions. We especially design a multi-layer representation based on scene graphs that encapsulates four different dimensions of knowledge. To exploit this new representation, we introduce a new large-scale dataset for indoor service robots with high-quality scene graph annotations. We then argue for the opportunities of using this representation to easily extract a wide range of fine-grained information about human interaction with context (All data and code are available at https://github.com/Maelic/IndoorVG).