This paper proposes to study the stability analysis and the self-excited vibration of a damped system of three degree-of-freedom. The proposed system is composed of a mass in frictional contact with a rigid plane moving with a constant rectilinear velocity. The frictional contact is modeled by the nonlinear and non-regular laws of unilateral contact and Coulomb's friction with a constant coefficient of friction. First of all, a complex eigenvalues analysis is conducted in order to evaluate the effects of various system parameters. Special attention will be given to the understanding of the role of damping and the associated destabilization paradox in mode-coupling instabilities with planar friction. Secondly, a shooting method is developed in order to determine the self-excited vibrations of the mechanical system. Optimized additional initial conditions based on an energetic method are also introduced. The last part of the study is devoted to the presentation and discussion of the results. The effectiveness of the proposed nonlinear approach is illustrated by a comparison with the results obtained by using a classical direct integration algorithm. Various nonlinear self-excited vibrations are observed and an evaluation of computational performances is also proposed.