Background: Efficacy and safety are the two considerations when characterising the effects of a new therapy. We sought to apply an innovative method of assessing the benefit-risk balance using data from a completed randomised controlled trial that compared erlotinib vs placebo added to gemcitabine in patients with advanced pancreatic cancer (NCIC CTG PA.3). Methods: We applied generalised pairwise comparisons with several prioritised outcome measures (e.g., one or more benefit outcomes and one or more risk outcomes). Here, the first priority outcome was overall survival (OS) time. Differences in OS that exceeded 2 months were considered clinically meaningful. The second priority outcome was toxicity. The overall treatment effect was quantified using the proportion in favour of erlotinib, which can be interpreted as the net proportion of patients who have a better overall outcome with erlotinib as compared with placebo. Sensitivity analyses were performed. Results: In this trial 569 patients were randomly assigned in a 1 : 1 ratio to receive gemcitabine plus either erlotinib or a matched placebo. Overall, the method indicated no statistically significant overall treatment effect in favour of erlotinib; if anything, the point estimate of the net proportion leaned in favour of the placebo group (overall proportion in favour of erlotinib = - 3.6%, 95% CI, - 14.2-7.1%; P = 0.51). The net proportion was never in favour of the erlotinib group throughout all sensitivity analyses. Conclusions: Generalised pairwise comparisons make it possible to assess the benefit-risk balance of new treatments using a single statistical test for any number of prioritised outcomes. The benefit-risk assessment was not in favour of adding erlotinib to gemcitabine for the treatment of patients with advanced pancreatic cancer.