Background: Rheumatoid arthritis (RA) is characterized by increased cardiovascular (CV) mortality. CV events are particularly high in patients with RA-specific autoimmunity, including rheumatoid factor (RF) and anti-citrullinated protein antibodies (ACPA), raising the question whether RA-specific autoimmunity itself is associated with CV events. Methods: New CV events (myocardial infarction, stroke or death by CV cause) were recorded in 20,625 subjects of the Electricité de France – Gaz de France (GAZEL) cohort. Self-reported RA cases in the GAZEL cohort were validated by phone interview on the basis of a specific questionnaire. In 1618 subjects, in whom plasma was available, RF and ACPA were measured. A piecewise exponential Poisson regression was used to analyze the association of CV events with presence of RA as well as RA-specific autoimmunity (without RA). Results: CV events in GAZEL were associated with age, male sex, smoking, hypertension, hyperlipidemia, and diabetes mellitus (HR from 1.06 to 1.87, p < 0.05). Forty-two confirmed RA cases were identified. Confirmed RA was significantly associated with CV risk increase (HR of 3.03; 95% CI: 1.13–8.11, p = 0.03) independently of conventional CV risk factors. One hundred seventy-eight subjects showed RF or ACPA positivity without presence of RA. CV events were not associated with ACPA positivity (HR: 1.52, 95% CI: 0.47–4.84, p = 0.48) or RF positivity (HR: 1.15, 95% CI: 0.55–2.40, p = 0.70) in the absence of RA. Conclusions: RA, as a clinical chronic inflammatory disease, but not mere positivity for RF or ACPA in the absence of clinical disease is associated with increased CV risk.