The interaction between Members of Parliament (MPs) is convention-based and rule-regulated. As instantiations of individual and group confrontations, parliamentary debates display well-regulated competing discursive processes. Unauthorised interruptions are spontaneous verbal reactions of MPs who interrupt the current speaker. This paper focuses on the answers of the current speaker to these disruptions. It introduces an annotation scheme for a political debate dataset which is mainly in the form of video annotations and audio annotations. The annotations contain information ranging from general linguistic to domain specific information. Some is annotated with automatic tools, and some is manually annotated. One of the goals is to use the information to predict the categories of the answers by the speaker to the disruptions. A typology of such answers is proposed and an automatic categorization system based on a multimodal parametrization is successfully performed.