This paper presents a recently-started doctoral project towards a new framework, called Virtual Intelligent Musical Objects (VIMOs). VIMOs aim at combining features of interactive machine learning tools, autonomous agents, and collaborative media into the creation of motion-based, user adaptable, shareable interactive music systems. We propose two models for stylistic motion learning and generation under a " design through performance " interactive work-ow. We discuss further human and computer-related research challenges involving expressiveness rendering and social musical interaction, as well as novel artistic and educational applications to be led within the scope of VIMOs.