In the context of a remote collaboration task in virtual reality, this study aimed to analyze the effects of task distribution on the processing of spatial information and mental workload in spatial dialogs. Pairs of distant participants with specific roles (a guide and a manipulator) had to collaboratively move a virtual object in a plane factory mock-up. The displays allowed the participants to be immersed together in the virtual environment. We analyzed the dialogs that took place according to the frames of reference and the mental transformations required to produce the spatial statements. We also measured the associated mental workload. Results showed that when participants took a perspective, the manipulator’s point of view was preferred. Perspective-taking only yielded a moderate increase in mental rotations, which may explain a specifically high mental demand score for the guides’ NASA-TLX. Overall, this is in accordance with the least collaborative effort principle. This study reinforces the idea that, in collaboration, operators do not need the same aids as each other. Thus, it is not necessary to develop symmetrical tools, i.e., the same tools for all co-workers; instead, the needs of each operator should be taken into account, according to the task he has to perform. In our case, the guides would be helped with perspective-taking aids, while the manipulators would be helped with action-oriented tools.