We propose a general framework to combine multiple sequence classifiers working on different sequence representations of a given input. This framework, based on Multi-Stream Hidden Markov Models (MS-HMMs), allows the combination of multiple HMMs operating on partially asynchronous information streams. This combination may operate at different levels of modeling: from the feature level to the post-processing level. This framework is applied to on-line handwriting word recognition by combining temporal and spatial representation of the signal. Different combination schemes are compared experimentally on isolated character recognition and word recognition tasks, using the UNIPEN international database.