Intermediate Evaluation Results from Living Labs
Résumé
This document demonstrates the intermediate evaluation process of the MTME based on the progress of the Herøya and Malpensa living labs. The focus is on needs for revisions and improvements before the next iteration. The main results include:
Hybrid research methods and data collection methods are proposed to evaluate the Multimodal Traffic Management Ecosystem in Orchestra based on the Program Theory evaluation and Science Design Research methodologies. We recall here that the Program Theory was chosen as the generic evaluation approach in the Orchestra project (more information is provided in D6.1, Evaluation handbook.
The evaluation approach includes three types of evaluation, effect evaluation, impact assessment and process evaluation that will be finally presented in the next deliverable, D6.3: Evaluation outcome.
The assessment is defined with 12 Key Performance Areas (KPAs) and corresponding research questions. The assessment is two folded: (1) a qualitative evaluation based on literature studies, interviews, workshops and (2) a quantitative evaluation based on Key Performance Indicators (KPIs).
The research questions have been verified ensuring their relevance and reliability for the evaluation of the MTME. Revised research questions have been proposed and the questions have been further validated by the different partners responsible for WPs and deliverables in order to find which tasks and deliverables will answer these questions in the final evaluation.
A combination of various research approaches is also defined per type of assessment based on the nature of the evaluation task.
The final list of KPIs and detailed data collection and analysis are elaborated in collaboration with WP4 (developing the enabling tools) and WP5 (simulation studies and living lab experiments).
Intermediate results on the stakeholders' acceptance are also summarised based on information collected through interviews, workshops, and literature studies.
In Appendix A, the data needed as input parameters into the simulations and real-life experimental studies of the 2 living labs are described and analysed. The data analysis describes quantitatively the current situation of passenger and freight transportation of the two living labs.
This intermediate evaluation is the first phase of Task 6.2 Evaluation, providing feedbacks and revisions of modelling, simulations, and experimental choices and setups. The final quantitative evaluation results will be obtained and described in D6.3: Evaluation outcome reporting the simulation and real-life proof-of-concept studies. Meanwhile D6.5: Recommendations and lessons learned will summarize the recommendations and lessons learned based on the results of the evaluation.