The aim of this paper is to develop a full Finite Element (FE) computational method for the dynamics of unstable frictional rolling contact systems in order to calculate reference solutions in many applications, especially curve squeal for railway transportation but also roller bearing or metal cutting. The proposed method is characterized by the use of a fine FE discretization of the contact surface in a Eulerian frame, nonlinear frictional contact laws and model reduction techniques. An application to the frictional rolling contact between two annular cylinders is presented in both quasi-static and dynamic cases with mode coupling instabilities. The validation of the approach in quasi-static conditions is carried out by comparison with CONTACT software. Stability and transient results show that the technique is able to simulate friction-induced vibrations at high frequencies. Reduced models are tested and show a good agreement with the full model.