Prediction of resonance in structures prone to unilateral contact is of high concern in the maintenance of structural health. For such structures, occurrences of resonance can be predicted by nonsmooth modes of vibration (NSMs). NSMs are families of periodic motions of the unforced and undamped structure and are characterized by a continuum of frequencies which are indicative of resonant frequencies. Here, a novel treatment of boundary conditions in the framework of Finite Element Method (FEM) allows for the detection of NSMs. Specifically, the motion of a bar in unilateral contact with a rigid wall is investigated. The wave equation is used to describe the dynamics of the bar while the unilateral contact setting is described using Signroini complementarity conditions. In turn, finding the NSMs of the bar in unilateral contact requires finding periodic solutions to a partial differential equation constrained by complementarity conditions. Commonly, the FEM has been used for numerical solutions of Signorini problems. In the FEM formulation, the Signorini problem is ill-posed and must be complemented with an impact law. Results show that, for an energy-conservative scheme, application of FEM with Newton impact law results in questionable chattering while a scheme allowing theoretical sticking at contact is energy dissipative. Thus, the impact law forbids periodic solutions with theoretical sticking. The proposed Nodal Boundary method allows for removing the impact law. Instead, the velocity of the contact node is determined by the FE stress approximation. Application of the proposed method results in a non-smooth ODE describing a switching between free and sticking-contact phases with the elimination of chattering at contact. Moreover, the scheme does not always dissipate energy and can exhibit periodic solutions. Using the harmonic balance method, it was possible to find periodic solutions for the problem of the bar of varying area in unilateral contact, a problem yet to be solved in literature.