Background Understanding social interactions requires the ability to accurately interpret conspecifics' actions, sometimes only on the basis of subtle body language analysis. Here we address an important issue that has not yet received much attention in social neuroscience, that of an interaction between two agents. We attempted to isolate brain responses to two individuals interacting compared to two individuals acting independently. Methodology/Principal Findings We used minimalistic point-light displays to depict the characters, as they provide the most straightforward way to isolate mechanisms used to extract information from motion per se without any interference with other visual information. Functional magnetic resonance imaging (fMRI) method was used to determine which brain regions were recruited during the observation of two interacting agents, mimicking everyday social scenes. While the mirror and mentalizing networks are rarely concurrently active, we found that both of them might be needed to catch the social intentions carried by whole-body motion. Conclusions/Significance These findings shed light on how motor cognition contributes to social cognition when social information is embedded in whole-body motion only. Finally, the approach described here provides a valuable and original tool for investigating the brain networks responsible for social understanding, in particular in psychiatric disorders.