Efficient perspective taking abilities are critical for successful social interactions. The aim of this study was to investigate whether and to what extent we spontaneously adopt others' perspectives in interpreting a visual scene. Through four distinct studies, we tested whether another person's point of view is adopted when interpreting an ambiguous stimulus. In addition, we manipulated the relevance of another person's perspective to a stimulus by altering sensory and action capabilities. The vast majority of participants spontaneously endorsed the other perspective rather than their own when asked to identify the target stimulus. Strikingly, and contrary to our expectations, they also overwhelmingly took the other's perspective when it was not relevant visually (blindfolded person) or behaviourally (armless person). A smaller number of subjects also adopted an external perspective in the absence of a person. Additionally, two control experiments show that viewing an injured person induces more compassion but does not trigger more perspective taking than viewing a healthy person. Our results demonstrate an overwhelming human tendency to predominantly consider non-egocentric perspectives while interpreting the surrounding world.