We aimed to analyze inequalities in social isolation among older adults in a time of COVID-19 social restrictions, using a gender perspective. A random population-based survey, including 21,543 older adults (65+) was conducted during and post COVID-19 lockdown in France. Our main outcome was a three-dimension indicator of social isolation based on living conditions, i.e., living alone (i) and not having gone out in the past week (ii), completed by an indicator measuring Internet use i.e., never using the Internet (iii). Logistic regressions were used to identify factors associated with isolation for women and men. Women were more likely to live alone (aOR = 2.72 [2.53; 2.92]), not to have gone out in the past week (aOR = 1.53 [1.39; 1.68]), and not to use the Internet (aOR = 1.30 [1.20; 1.44]). In addition to gender effects, being older, at the bottom of the social hierarchy, and from an ethno-racial minority was also associated with social isolation. Preventive policies should take into account these inequalities when addressing the issue of social isolation among older women and men, so as to enable all social groups to maintain social contacts, and access health information.