As artificial intelligence (AI) is increasingly present in different aspects of society and its harmful impacts are more visible, concrete methods to help design ethical AI systems and limit currently encountered risks must be developed. Taking the example of a well-known Operations Research problem, the Nurse Rostering Problem (NRP), this paper presents a way to help close the gap between abstract principles and on-the-ground applications with two different steps. We first propose a normative step that uses dedicated scientific knowledge to provide new rules for an NRP model, with the aim of improving nurses’ well-being. However, this step alone may be insufficient to comprehensively deal with all key ethical issues, particularly autonomy and explicability. Therefore, as a complementary second step, we introduce an interactive process that integrates a human decision-maker in the loop and allows practical ethics to be applied. Using input from stakeholders to enrich a mathematical model may help compensate for flaws in automated tools.