Immersive virtual reality systems or high resolution wall-sized displays become more common for analyzing the increasing among of data from science, industry, business and society. These large interactive spaces are powerful tools to enable remote users to work together on shared data. However, we cannot imagine that remote collaboration in such systems becomes widespread if it requires that all users have the exact same physical devices, and we need to make remote collaboration across different systems possible. The asymmetric interaction capabilities of each user are also an interesting opportunity to develop new collaboration strategies. In this short position paper, we present our past work about collaborative virtual environments and, in particular, how to represent the physical environments of each user in a virtual environment. We also introduce an ongoing project which aims to support remote collab-orative interaction across heterogeneous large interactive spaces.