Research paper recommender systems (RSs) aim to alleviate information overload for researchers. Existing approaches using collaborative filtering or hybrid approaches typically allow only one rating criterion (overall liking) for users to evaluate papers. We conducted a focus group qualitative study to explore the most important criteria for rating research papers that can be used to control the paper recommendation by enabling users to set the weight for each criterion. We investigate also the effect of using different rating criteria on the user interface design and how the user can control the weight of the criteria.