Unsupervised 3D pose estimation has gained promi-nence due to the challenges in acquiring labeled 3D data fortraining. Despite promising progress, unsupervised approachesstill lag behind supervised methods in performance. Two factorsimpede the progress of unsupervised approaches: incompletegeometric constraint and inadequate interaction among spatial,temporal, and multi-view features. This paper introduces anunsupervised pipeline that uses calibrated camera parametersas geometric constraints across views and coordinate spaces tooptimize the model by minimizing inconsistencies between the2D input pose and the re-projection of the predicted 3D pose.This pipeline utilizes the novel Hierarchical Cross Transformer(HCT) to encode higher levels of information by enabling in-teractions among hierarchical features containing different levelof temporal, spatial and cross-view information. By minimizingthe reliance on human-specific parts, the HCT shows potentialfor adapting to various pose estimation tasks. To validate theadaptability, we build a connection between human pose estima-tion and scene pose estimation, introducing Dynamic-Keypoints-3D (DK-3D) dataset tailored for 3D Scene Pose Estimationin robotic manipulation. Experiments on two 3D human poseestimation datasets demonstrate our method’s new state-of-the-art performance among weakly and unsupervised approaches.The adaptability of our method is confirmed through experimentson DK-3D, setting the initial benchmark for unsupervised 2D-to-3D scene pose lifting.