A large body of literature has explored the rise of extreme right parties since the 1980s in Europe. A number of studies have assessed whether the context, particularly the level of unemployment and immigration, impacts support for the extreme right and found contradictory results. We argue in this paper that these contradictions might simply result from differences in the scale used to assess the context. We systematically test the impact of immigration and unemployment, measured at various scales, on extreme right voting and attitudes towards immigrants and immigration, using two alternative approaches with data for Belgium. In the concentric approach, we measure the unemployment and the presence of migrants within various radii around each respondent's residence. In the polarized approach, we consider the actual polarization of space, using the very local scale (statistical district), the living pool (municipality), and the employment pool. We show that the scale choice changes the results and that the most significant impacts are generally found at intermediary scales.