This paper aims at questioning what Sen (2009) presents as a theory of justice derived from Smith’s idea of the “impartial spectator”. Sen’s tribute to Smith’s pioneering concept of the impartial spectator already gave rise to a set of criticisms that we divide in two kinds: 1) Sen’s reading is unfaithful with regard to the original Smithian concept (Forman-Barzilai, 2010; Gilardone, 2010; Bruni, 2011; Alean, 2014; Shapiro, 2011; Ege, Igersheim & Le Chapelain, 2012) and 2) Sen’s reading is a weak point of his theory of justice (Shapiro, 2011; Ege, Igersheim & Le Chapelain, 2012). In the paper, we try to address both kinds of criticism. Firstly, we shed a new light on Sen’s reading of Smith and provide a path of reconciliation between Sen’s analysis and Smith’s one. For us, Sen’s impartial spectator is somewhat reminiscent of another figure from Smith’s moral philosophy: “the man without”. Secondly, we show that, in Smith’s analysis, “the man without” is pointless without his genuine concept of the impartial spectator, called “the man within”. We conclude by arguing that Smith’s “man within” could constitute the missing piece in Sen’s analysis of the process which must lead public reasoning towards more justice. Introducing a missing touch of Smith could thus strengthen Sen’s idea of open impartiality in public reasoning for challenging Rawls’ contractualist theory of justice.