Quality assessment (QA) has long been considered essential to guarantee the reliability of neuroimaging studies. It is particularly important for fetal brain MRI, where unpredictable fetal motion can lead to substantial artifacts in the acquired images. Multiple images are then combined into a single volume through super-resolution reconstruction (SRR) pipelines, a step that can also introduce additional artifacts. While multiple studies designed automated quality control pipelines, no work evaluated the reproducibility of the manual quality ratings used to train these pipelines. In this work, our objective is twofold. First, we assess the inter- and intra-rater variability of the quality scoring performed by three experts on over 100 SRR images reconstructed using three different SRR pipelines. The raters were asked to assess the quality of images following 8 specific criteria like blurring or tissue contrast, providing a multi-dimensional view on image quality. We show that, using a protocol and training sessions, artifacts like bias field and blur level still have a low agreement (ICC below 0.5), while global quality scores show very high agreement (ICC = 0.9) across raters. We also observe that the SRR methods are influenced differently by factors like gestational age, input data quality and number of stacks used by reconstruction. Finally, our quality scores allow us to unveil systematic weaknesses of the different pipelines, indicating how further development could lead to more robust, well rounded SRR methods.