Visual and haptic perceptions of 3D shape are plagued by distortions, which are influenced by non-visual factors, such as gravitational vestibular signals. Whether gravity acts directly on the visual or haptic systems or at a higher, modality-independent level of information processing remains unknown. To test these hypotheses, we examined visual and haptic 3D shape perception by asking male and female human subjects to perform a “squaring” task in upright and supine postures and in microgravity. Subjects adjusted one edge of a 3D object to match the length of another in each of the 3 canonical reference planes and we recorded the matching errors to obtain a characterization of the perceived 3D shape. The results show opposing, body-centered patterns of errors for visual and haptic modalities, whose amplitudes are negatively correlated, suggesting that they arise in distinct modality-specific representations that are nevertheless linked at some level. On the other hand, weightlessness significantly modulated both visual and haptic perceptual distortions in the same way, indicating a common, modality-independent origin for gravity's effects. Overall, our findings show a link between modality-specific visual and haptic perceptual distortions and demonstrate a role of gravity-related signals on a modality-independent internal representation of the body and peripersonal 3D space used to interpret incoming sensory inputs. Significance Statement Both visual and haptic 3D-object perception are plagued by anisotropic patterns of errors, as shown in a task of “squaring” the faces of an adjustable cube. We report opposing and negatively correlated perceptive errors for the visual and haptic perceptions, suggesting a strong interaction between the two sensory modalities, even when the task was fundamentally unimodal. In addition, the similar effect of microgravity observed on both visual and haptic perception indicates that gravity acts on a modality-independent representation of 3D space used to process these sensory inputs. These findings foster awareness that even simple, unimodal, egocentric tasks are likely to involve complex, cross-modal signal processing.