Article Dans Une Revue Law, Technology and Humans Année : 2024

Case Law as Data : Prompt Engineering Strategies for Case Outcome Extraction with Large Language Models in a Zero-Shot Setting

Résumé

This study explores the effectiveness of prompt optimization techniques for legal case outcome extraction using Large Language Models (LLMs). Two state-of-the-art LLMs, LLaMA3 70b and Mixtral 8x7b, are used in a zero-shot data extraction task on a diverse dataset of 400 French appellate court decisions. The results show that LLMs exhibit remarkable efficiency in extraction tasks. Our findings indicate that baseline prompts achieve high performance metrics, with a best F1 score of 0.980 and a worst F1 score of 0.853. Optimized prompts yield varying degrees of improvement, with a best F1 score of 0.994 and a worst F1 score of 0.912. While some optimized prompts demonstrate significant improvements, others exhibit minor or even negative changes. Our results suggest that the optimization process has a non-uniform impact on performance metrics, and the effectiveness of optimized prompts depends on the specific model and dataset being used. These results underscore the significance of prompt engineering in optimizing LLM performance for Legal Information Extraction and Litigation Analytics reliability.

Fichier principal
Vignette du fichier
3623-Article Text-14089-1-10-20241125.pdf (1.05 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence
DOI

Cite 10.17605/OSF.IO/M4FKT Article ZAMBRANO, G. (2024). LILA: Litigation Data Extraction with LLMs. OSF. https://doi.org/10.17605/OSF.IO/M4FKT

Dates et versions

hal-04870063 , version 1 (07-01-2025)

Licence

Identifiants

Citer

Guillaume Zambrano. Case Law as Data : Prompt Engineering Strategies for Case Outcome Extraction with Large Language Models in a Zero-Shot Setting. Law, Technology and Humans, 2024, 6 (3), pp.80 - 101. ⟨10.5204/lthj.3623⟩. ⟨hal-04870063⟩

Collections

86 Consultations
330 Téléchargements

Altmetric

Partager

  • More