Digital (or digitized) documents have become a prominent material for a lot of research fields. As such, researchers need tools to enrich those documents, both to support the analytical process and to render the results of their work. The enrichment can come from automatized, AI powered processes, as well as humans in a close reading approach. The goal is to make the original content more explicit by annotating it and linking it to a multimedia context. Such pieces of software can be characterized as multimedia and multi-modal annotation tools. As shown in the state of the art for video annotation tools (Bardiot et al. 2023), they can differ greatly regarding their supported formats, annotation modalities, cooperation features, sharing capabilities, interoperability, extensibility, etc. We believe that extensibility and interoperability are the truly key features to evaluate those tools, as illustrated in initiatives such as (Amato et al. 2022). While each of these many tools offer unique perspectives and cutting-edge features, there is a real danger of vendor lock-in for the communities that use them. This leads to the fragmentation of annotated content in silos, only exploitable with the tools used for their respective creation. This lack of interoperability prevents nearly any kind of collaborative and incremental research. Without resolving all of these issues entirely, we believe that Open Source tools, built on standard formats and API, are a sound way to ensure interoperability, re-usability of annotated content and extensibility with new features. As such, we present our ongoing effort to develop MemoRekall-IIIF (consult the live demos and documentation at https://memorekall.com/en/memorekall-iiif-prototype/), a platform that aims at providing an extensible software base for all communities in need for multimedia and multi-modal annotation tools. It is modular, Open Source and built on the IIIF standard (https://iiif.io/). MemoRekall-IIIF is a complete re-implementation of MemoRekall (https://memorekall.com/, Bardiot 2021), a video annotation tool developed in a cocreation process between developers and digital humanities researchers. The software currently takes the form of a fork of the widely- used IIIF-viewer Mirador (https://projectmirador.org/). We have notably added video and video annotation features to the viewer. We build upon this tool to propose an environment for the creation of multi-modal document networks. Each document is presented as a IIIF manifest: a lightweight, interoperable file that is interpreted by the viewer to display the associated media and annotations. An annotation can be linked to another IIIF manifest and opened directly within the environment, providing a fluid and dynamic navigation experience. While retaining many of the features of its predecessor MemoRekall (and targeting full retro compatibility), MemoRekall-IIIF proposes several notable technical and epistemological shifts: most significantly a move away from the necessary focus on a single video towards a document network perspective. The environment is being designed with extensibility in mind, leaving it open to the incremental addition of processing modules for the automated creation of annotations and network analysis. Our first phase of analysis was built upon a concrete case-study, and going forward we intend to continue the practice-led design philosophy that was so successful for MemoRekall with more case-studies and workshops with various user types.