Building a Distributed Computing Network for Galaxy, Application to Genome Annotation
Building a Distributed Computing Network for Galaxy, Application to Genome Annotatio
Résumé
As an open-source web application, Galaxy enables biologists and bioinformaticians to perform scientific calculations (simple or complex) on different computing resources (clusters, clouds). Galaxy offers a user-friendly interface, while enabling the processing of large volumes of data (in the terabyte range) through complex processing chains. Galaxy simplifies the rapid creation of complex and reproducible pipelines while committing to addressing the specific needs of diverse research communities. Supported by a vibrant community, Galaxy's services continue to expand and evolve, ensuring its adaptability. Galaxy is well adopted in the French ecosystem with 10+ servers and UseGalaxy.fr, the French UseGalaxy server.
EuroScienceGateway (ESG) is a project (2023-2025) set up under the European Union programme Horizon Europe to facilitate access to computing and storage infrastructures across Europe in line with researchers’ needs. The main aim is to provide a robust, scalable and fully integrated open infrastructure, based on Galaxy, for analysing data from a wide range of scientific fields. The GenOuest platform is a partner of the ESG project, in close collaboration with the Institut Français de Bioinformatique (IFB) and its UseGalaxy.fr portal.
EuroScienceGateway infrastructure
The ESG infrastructure relies on UseGalaxy.eu (the european Galaxy server) and 7 national UseGalaxy.* servers, including UseGalaxy.fr, the French server supported by the Institut Français de Bioinformatique (IFB). These servers provide access to local computing resources, based on classic scheduling systems (e.g. Slurm, HTCondor, …). One of the big aims of ESG is to deploy a network of remote computing endpoints (12 currently), using the Pulsar software, and to make these computing resources usable by each of the UseGalaxy.* servers.
The deployment of a Pulsar endpoint is based on Terraform, an "infrastructure as code" system, which enables the automated deployment of different interconnected virtual machines (VM) within an OpenStack cloud. In the case of a Pulsar endpoint, it deploys a set of VMs: a computing cluster based on HTCondor with a central manager (running the Pulsar daemon) and several compute nodes, and an NFS shared volume accessible from all VMs. Each of these VMs are run using a preconfigured image called "virtual galaxy compute nodes" (https://github.com/usegalaxy-eu/vgcn). This source image contains all the components needed to run standard Galaxy jobs. A first French Pulsar endpoint (FR01) has been successfully deployed on the GenOuest infrastructure using this mechanism.
Communication between a UseGalaxy.* server and a Pulsar endpoint is achieved using a RabbitMQ message queuing system deployed alongside Galaxy. Through this system, Galaxy and Pulsar can communicate tasks to be executed, track their status and exchange results and metadata easily and robustly. From the Galaxy point of vue, a Pulsar endpoint is considered just as any other scheduling system (e.g. Slurm, HTCondor, …) and jobs can be routed to it easily using the recently developed Total Perspective Vortex (TPV). A typical Pulsar job execution will trigger the transfer of input data to the remote computing infrastructure, the execution of the remote job, then the retrieving of the results.
EuroScienceGateway use case: genome annotation
Four use cases have been identified in ESG to demonstrate the effectiveness of the distributed computing infrastructure: Astronomy, Climate, Biodiversity and Material Science. GenOuest is contributing to the Biodiversity use case by implementing a large collection of tools (Braker3, Helixer, Compleasm, OMark, …), workflows (https://workflowhub.eu/collections/13) and training material for the annotation of genomes. These developments are carried out in collaboration with large scale genome sequencing projects under the EBP (Earth BioGenome Project) umbrella, in particular VGP (Vertebrate Genomes Project). This initiative aims to generate virtually error-free reference genomes for vertebrate species, crucial for addressing key questions in biology and pathology, identifying species at highest risk of genetic extinction, and safeguarding biodiversity genetic information. Consequently, tailored workflows have been designed, and benchmarked on a first batch of 16 genomes, with varying combinations of repeat masking and annotation methods.
Perspectives
The developments achieved in the ESG project bring exciting perspectives for the future, including smarter meta-scheduling mechanisms allowing to:
reduce CO2 emissions (by moving jobs to more efficient facilities)
use specific computing architecture (e.g. by running jobs in facilities providing GPUs)
execute jobs as close as possible to input data (will eliminate the need for data transfers)
make computing resources pluggable even by users (BYOC: Bring Your Own Compute)
Origine | Fichiers produits par l'(les) auteur(s) |
---|