(1) Background: Several smaller studies have shown that COVID-19 patients with cancer are at a significantly higher risk of death. Our objective was to compare patients hospitalized for COVID-19 with cancer to those without cancer using national data and to study the effect of cancer on the risk of hospital death and intensive care unit (ICU) admission. (2) Methods: All patients hospitalized in France for COVID-19 in March–April 2020 were included from the French national administrative database, which contains discharge summaries for all hospital admissions in France. Cancer patients were identified within this population. The effect of cancer was estimated with logistic regression, adjusting for age, sex and comorbidities. (3) Results: Among the 89,530 COVID-19 patients, we identified 6201 cancer patients (6.9%). These patients were older and were more likely to be men and to have complications (acute respiratory and kidney failure, venous thrombosis, atrial fibrillation) than those without cancer. In patients with hematological cancer, admission to ICU was significantly more frequent (24.8%) than patients without cancer (16.4%) (p < 0.01). Solid cancer patients without metastasis had a significantly higher mortality risk than patients without cancer (aOR = 1.4[1.3–1.5]), and the difference was even more marked for metastatic solid cancer patients (aOR = 3.6[3.2–4.0]). Compared to patients with colorectal cancer, patients with lung cancer, digestive cancer (excluding colorectal cancer) and hematological cancer had a higher mortality risk (aOR = 2.0[1.6–2.6], 1.6[1.3–2.1] and 1.4[1.1–1.8], respectively). (4) Conclusions: This study shows that, in France, patients with COVID-19 and cancer have a two-fold risk of death when compared to COVID-19 patients without cancer. We suggest the need to reorganize facilities to prevent the contamination of patients being treated for cancer, similar to what is already being done in some countries.