The activity of part orientation is of crucial importance in terms of impact on the quality of parts in powder bed additive manufacturing. To support the decision-making of Computer Aided Manufacturing (CAM) operators, several knowledge and software models are proposed. Even if they seem to help operators in orientation process, their operational effectiveness still needs to be scientifically assessed. This paper proposes to evaluate the efficiency of a part orientation software in assisting CAM engineers in part orientation choice. The related software is based on the mathematization of action rules issued from a knowledge model. The knowledge involved to carry out the orientation activity is analyzed based on the users' expertise. To do this, an empirical approach based on a case study is made with participants of different levels of skills. Several design scenarios including various part typologies to be oriented are submitted to engineers, before their manufacturing with Electron Beam Melting (EBM) technology. Two means of orientation are thus used for comparison, namely a manual orientation and a computer-aided orientation software. Based on the orientation results, an analysis of the software usage is undertaken. As an underlying result of our study, we have come up with an evaluation approach that can be reused in other contexts and with other software.