In this work, we study collective intelligence behavior of Web users that share and watch video content. We propose that the aggregated users’ video activity exhibits characteristic patterns that may be used in order to infer important video scenes thus leading to collective intelligence concerning the video content. In particular, we have utilised a controlled user experiment with information-rich videos for which users’ interactions (e.g., pause, seek/scrub) have been gathered. Modeling the collective information seeking behavior by means of the corresponding probability distribution function we argue that bell-shaped reference patterns are shown to significantly correlate with the predefined scenes of interest for each video, as annotated by the users. In this way, the observed collective intelligence may be used to provide a video-segment ranking tool that detects the importance of video scene. In practice, the proposed techniques might improve navigation within videos on the web and have also the potential to improve video search results with personalised video thumbnails.