We describe the design of a system consisting of several state-of-the-art real-time audio and video processing components enabling multimodal stre am manipulation (e.g., automatic online editing for multiparty videoconferencing applications) in open, unconstrained environments. The underlying algorithms are designed to allow multiple people to enter, interact, and le ave the obser vable scene with no constraints. They comprise continuous localisation of audio objects and its application for spatial audio object coding, detection, and tracking of faces, estimation of head poses and visual focus of attention, detection and localisation of verbal and paralinguistic events, and the association and fusion of these different events. C ombined all together, they represent multimodal streams with audio objects and semantic video objects and provide semantic information for stream manipulation systems (like a virtual director). Various experiments have b een performed to evaluate the performance of the system. The obtained results demonstrate the effectiveness of the proposed design, the various a lgorithms, and the b enefit of fusing different modalities in this scenario.