Pré-Publication, Document De Travail Année : 2025

Are Autonomous Web Agents Good Testers?

Antoine Chevrot
  • Fonction : Auteur
  • PersonId : 1223910
  • IdRef : 265867630
Alexandre Vernotte
  • Fonction : Auteur
Bruno Legeard
  • Fonction : Auteur

Résumé

Despite advances in automated testing, manual testing remains prevalent due to the high maintenance demands associated with test script fragility-scripts often break with minor changes in application structure. Recent developments in Large Language Models (LLMs) offer a potential alternative by powering Autonomous Web Agents (AWAs) that can autonomously interact with applications. These agents may serve as Autonomous Test Agents (ATAs), potentially reducing the need for maintenance-heavy automated scripts by utilising natural language instructions similar to those used by human testers. This paper investigates the feasibility of adapting AWAs for natural language test case execution and how to evaluate them.

We contribute with (1) a benchmark of three offline web applications, and a suite of 113 manual test cases, split between passing and failing cases, to evaluate and compare ATAs performance, (2) SeeAct-ATA and pinATA, two open-source ATA implementations capable of executing test steps, verifying assertions and giving verdicts, and (3) comparative experiments using our benchmark that quantifies our ATAs effectiveness. Finally we also proceed to a qualitative evaluation to identify the limitations of PinATA, our best performing implementation.

Our findings reveal that our simple implementation, SeeAct-ATA, does not perform well compared to our more advanced PinATA implementation when executing test cases (50% performance improvement). However, while PinATA obtains around 60% of correct verdict and up to a promising 94% specificity, we identify several limitations that need to be addressed to develop more resilient and reliable ATAs, paving the way for robust, low maintenance test automation.

CCS Concepts: • Software and its engineering → Software testing and debugging.

Fichier principal
Vignette du fichier
main_camera_ready.pdf (1.67 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05014439 , version 1 (01-04-2025)

Licence

Identifiants

Citer

Antoine Chevrot, Alexandre Vernotte, Jean-Rémy Falleri, Xavier Blanc, Bruno Legeard. Are Autonomous Web Agents Good Testers?. 2025. ⟨hal-05014439⟩
232 Consultations
391 Téléchargements

Altmetric

Partager

  • More