The aim of tolerance analysis is to find a balance between manufacturing cost and product geometric quality. Earlier research usually considered ideal surface-based deviation models to conduct simulations, which ignore form defects. With the increasing demands on quality control, a skin model shape that includes form defects has its advantages. First, the effects of the chosen geometrical model on the assembly simulation are analyzed; the geometrical models considered are: nominal model, ideal surface-based deviation model, and skin model shape. The importance of form defects and assembly load conditions is highlighted. However, it is not easy to integrate all these factors into tolerance analysis. Difficulties in assembly simulation are discussed, such as the mating between non-ideal surfaces, model balance under external and internal loads, the consistency of simulation results, etc. Based on the analysis, a skin model shape-based tolerance analysis method, which also considers the load and displacement boundary conditions, is proposed. The assembly of the skin model shape is transformed into the objective function of a quadratic optimization problem, links between physical properties and optimization constraints are established. Corresponding simulation results are generated by conducting the optimization iteratively. To illustrate the method and validate its simulation result, various models are used as examples. Lastly, a simplified hand saw model is used as a case study.