Modern societies face many challenges, one of them is the rise of affective polarization over the last 4 decades. In an attempt to understand its reasons, many researchers have questioned the role of Social Media in general, and Recommender Systems (RS) in particular, on the emergence of these extreme behaviors. Diversity in News Recommender Systems (NRS) was quickly perceived as a major issue for the preservation of a healthy democratic debate. However, after more than 15 years of research in Artificial Intelligence on the subject, the understanding of the real impact of diversity in recommendations remains limited. Through a case analysis on the well-known MIND dataset, we propose a critique of the diversity-aware recommendation and evaluation approaches, and provide some take-home messages related to the need of adapted datasets, diversity metrics and analytical methodologies.