Mining experimental data from Materials Science literature with Large Language Models: an evaluation study - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

Résumé

This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.
Fichier principal
Vignette du fichier
Mining_experimental_data_from_Materials_Science_literature_with_Large_Language_Models-clean.pdf (1.01 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04407016 , version 1 (20-01-2024)
hal-04407016 , version 2 (09-04-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04407016 , version 2

Citer

Luca Foppiano, Guillaume Lambard, Toshiyuki Amagasa, Masashi Ishii. Mining experimental data from Materials Science literature with Large Language Models: an evaluation study. 2024. ⟨hal-04407016v2⟩
11 Consultations
19 Téléchargements

Partager

Gmail Facebook X LinkedIn More