In 3D reconstruction application domains, for example, repairing relics, bone fracture surgery, the reassembly of fragmented objects is required. Intensive research has focused on virtual 3D reconstruction but very little attention was paid to the physical assembly process where original segments are dealt with. To obtain a good reassembly result with reduced damage risk, assembly error and operation time, it is not enough to only use a 3D virtual model or a physical prototype model as assembly reference. The assembly sequence and the assembly operation should be investigated. To support the physical reconstruction, an integrated method which uses both virtual and physical prototyping technologies with human interaction is proposed. This paper mainly discusses the physical assembly sequence optimisation which is the partial implementation of the proposed method. An experimental pilot case study is presented to demonstrate the importance and potential of the method.