Towards an open ETL Framework to support studies on mountain land sharing
Résumé
Moutain natural areas are the stage for diverse wildlife and human usage that often conflict. In the past years, outdoor leisure communities have particularly grown, leading to ecological pressure on wildlife. Scientists use different types of data to understand these ecosystems and elaborate recommendations to achieve sustainable ecosystems where land is shared. They spend a lot of time engineering data properly before being able to produce new knowledge. The IntForOut project reunites ecologists and geodata scientists to use data from various sources to study human impact on mountain ecosystems in Mont Blanc and investigate innovative uses of stakeholder-generated data. We present the data engineering challenges encountered by ecologists and potential solutions investigated in the project. Challenges in discovering and integrating data relevant to alpine ecology studies Scientists studying alpine ecology use many data: hikers tracks, camera trap data, animals GPS tracks, habitat data computed by protected areas authorities, land cover and land use data, weather data, points of interest. These are different types of sources, publicly mandated authorities products, volunteered geography, crowd sourced data, scientific studies or private sector. The data is served from different portals, sometimes through a visualization interface only. The expertise required to use raw data, like for example the explicit meaning of terms like “acquisition campaign”, is not explicated online. Two highly important criteria for these scientists are : 1) the temporality of data, i.e. when was the data acquired and what is the temporal validity of it (which can be seasonal), when is it updated, 2) the sensitivity of data and the conditions under which the results can be published. Both criteria are hard to evaluate on current portals. The raw data must be transformed and integrated before analysis, for example computing the human frequentation for each segment of hiking trail, or human–wildlife coexistence places. This often necessitates database manipulation and can also require spatial analysis operations. Scientists encounter the following categories of challenges while deriving consistent data for their analysis : various temporal and spatial granularities across datasets, incomplete data, outdated data and different update mechanisms. Approaches developed in IntForOut project to integrate this data To address these issues, we define a dedicated open ETL (Extract, Transform, Load) process that assists in discovering, cleaning, updating, linking and documenting the different data used by these scientists. It consists in : - an IntForOut ontology that describe data –including underlying concepts- and integration methods, - a unified data warehouse, that can be replicated by scientists, - a metadata information infrastructure, where metadata is any data about the data at stakes: structured metadata, official documentation, project description, readme files, fragments in meetings minutes The next set of challenges involves defining a user interface to access the ETL operations, a generic process for integrating new data, as well as establishing an automated metadata generation workflow to document both the data and the associated transformation procedures.