Within the framework of structural dynamics, modal analysis is a practical tool to predict vibrations in order to prevent or utilize them. As a challenging topic in the field of modal analysis, nonlinear modal analysis is focused by many researchers [1]. Nonsmooth modal analysis is a special form of nonlinear modal analysis, in which governing equation of targeting system involves non-differentiable functions of the state. To perform non-smooth modal analysis, a method combing Frequency Domain Boundary Element Method (FD-BEM) and Harmonic Balance Method (HBM) is applied.