In this paper, we report on the evaluation methodology developed to support the experimental plan to assess the big data prototype developed under the umbrella of the datAcron project. The datAcron prototype was decomposed along several semantic levels of functionality: Maritime data, datAcron components, Maritime Situational Indicators (MSIs), and Maritime scenarios. Different experimental features have been selected for delivering representative results at all levels of system integration. The results were captured in an evaluation framework unifying the different evaluation criteria and measures into the following groups: (1) timeliness and scalability, (2) compression ratio, (3) classification quality and (4) clarity and effectiveness. By the three dimensions of system decomposition, big data variations and evaluation criteria an experimental space is opened, with respect to which all experiments are referenced. Major findings on the performance of the components of datAcron prototype are reported, together with other findings regarding the user assessment of the effectiveness and the clarity of the datAcron prototype.