Television is a medium that is implicitly highly codified. Every TV program has its own visual identity that is often rich in information; most of the time, a single frame extracted from a TV broadcast contains enough information for a human agent to determine the genre of the program, and sometimes even to predict who is likely to appear in it. Our goal is to exploit the visual context of TV programs to help identify the people appearing in them. In this work, we introduce a new dataset of over 10 M frames extracted mainly from french TV programs and aired between 2010 and 2020. We also present an original approach for deep similarity metric learning in order to learn a descriptor that effectively captures the visual context of a TV program and helps to recognize the subjects appearing in the program.