Pré-Publication, Document De Travail Année : 2025

An Expert-grounded benchmark of General Purpose LLMs in LCA

Résumé

Purpose: Artificial intelligence (AI), and in particular large language models (LLMs), are increasingly being explored as tools to support life cycle assessment (LCA). While demonstrations exist across environmental and social domains, systematic evidence on their reliability, robustness, and usability remains limited. This study provides the first expert-grounded benchmark of LLMs in LCA, addressing the absence of standardized evaluation frameworks in a field where no clear ground truth or consensus protocols exist. Methods: We evaluated eleven general-purpose LLMs, spanning both commercial and open-source families, across 22 LCA-related tasks. Seventeen experienced practitioners reviewed model outputs against criteria directly relevant to LCA practice, including scientific accuracy, explanation quality, robustness, verifiability, and adherence to instructions. We collected 168 expert reviews. Results: Experts judged 37% of responses to contain inaccurate or misleading information. Ratings of accuracy and quality of explanation were generally rated P R E -P R I N T "average" or "good" on many models even smaller models, and format adherence was generally rated favourably. Hallucination rates varied significantly, with some models producing hallucinated citations at rates of up to 40%. There was no clear-cut distinction between ratings on open-weight versus closed-weight LLMs, with open-weight models outperforming or competing on par with closed-weight models on criteria such as accuracy and quality of explanation. Conclusion: These findings highlight the risks of applying LLMs naïvely in LCA, such as when LLMs are treated as free-form oracles, while also showing benefits especially around quality of explanation and alleviating labour intensiveness of simple tasks. The use of general-purpose LLMs without grounding mechanisms presents quantifiable risks that can directly affect LCA result quality, such as hallucination of citations. This work underscore the need for larger, more diverse benchmarks to better quantify and improve standards around LLM use, including grounding mechanisms to lower the rate of occurrence of undesirable model behaviour. While no system can eliminate errors, development of robust testing standards offer clarity, and a pathway to improve and validate best practices when LLMs are used in LCA workflows.

Fichier principal
Vignette du fichier
Expert_benchmark_of_LLMs_for_LCA_tasks (preprint watermark).pdf (1.33 Mo) Télécharger le fichier
Methods v1.4.png (285.12 Ko) Télécharger le fichier
accuracy_per_model.png (62.64 Ko) Télécharger le fichier
barchart_accuracy_explanation.png (128.4 Ko) Télécharger le fichier
barchart_accuracy_per_llm.png (64.29 Ko) Télécharger le fichier
barchart_explanation_per_llm.png (67.5 Ko) Télécharger le fichier
barchart_format_per_llm.png (66.49 Ko) Télécharger le fichier
barchart_incorrect_per_llm.png (76.32 Ko) Télécharger le fichier
boxplot_citations_per_llm.png (37.56 Ko) Télécharger le fichier
explanation_per_model.png (67.42 Ko) Télécharger le fichier
format_correctness.png (50.36 Ko) Télécharger le fichier
hallucination_rate.png (84.91 Ko) Télécharger le fichier
heatmap_accuracy_and_explanation.png (70.25 Ko) Télécharger le fichier
heatmap_all_criteria.png (123.92 Ko) Télécharger le fichier
heatmap_all_criteria_corr.png (71.09 Ko) Télécharger le fichier
heatmap_by_task_type.png (114.89 Ko) Télécharger le fichier
pie_chart_expected_format.png (30.67 Ko) Télécharger le fichier
pie_chart_incorrect_misleading.png (35.47 Ko) Télécharger le fichier
survey_screenshot.png (89.75 Ko) Télécharger le fichier

Dates et versions

hal-05319796 , version 1 (17-10-2025)

Licence

Identifiants

  • HAL Id : hal-05319796 , version 1

Citer

Artur Donaldson, Bharathan Balaji, Cajetan Oriekezie, Kumar Manish, Laure Patouillard. An Expert-grounded benchmark of General Purpose LLMs in LCA. 2025. ⟨hal-05319796⟩
353 Consultations
151 Téléchargements

Partager

  • More