In multimedia quality assessment, observer ratings are typically averaged into mean opinion scores (MOS) to obtain a subjective ground truth for a set of stimuli. Valuable information about individual observer rating behaviour and interobserver differences is lost during this process. Such information, however, can be useful to improve subjective experiment procedures and quality of experience prediction models. In this paper, we therefore present an inter-observer analysis framework that tackles the quality assessment problem from an inverse angle, setting the focus on individual observer differences rather than stimuli differences. The framework consists of a set of procedures for inter-observer analysis as well as the necessary considerations during pre-processing and post-processing. The aim of this paper is to raise awareness that sole consideration of MOS simplifies quality assessment too much, especially given the ever increasing complexity of multimedia quality assessment.