A Cross-platform Investigation of Complexity for Russian Learners of English - Archive ouverte HAL
Communication Dans Un Congrès Année : 2021

A Cross-platform Investigation of Complexity for Russian Learners of English

Résumé

This paper discusses the automatic analysis of linguistic complexity, comparing the benefits of two existing NLP tools. Complexity is a construct which is part of the Complexity, Accuracy, Fluency (CAF) triad (Housen et al., 2012). Taken in its linguistic sense, the construct can be divided into two main categories. It may include systemic or structural aspects of language, each branching out into specific linguistic dimensions such as syntax and lexis (Norris & Ortega, 2009; Wolfe-Quintero et al., 1998). Linguistic complexity has been tested in many L2 tasks dedicated to proficiency prediction. It has been operationalised with several types of metrics such as the ubiquitous Type Token Ratio (TTR). Results have shown correlations between specific metrics and proficiency (Ballier et al., 2020; Crossley et al., 2011; Kyle, 2016; Vajjala, 2017). However, automatic analysis of CEFR learnel levels is not as simple a task and previous studies show differences in predictions between levels. The B level appears to be challenging as several systems report confusions between the two adjacent B1 and B2 levels. This is all the more serious that this distinction corresponds to the threshold level which is key for many language education institutions. A fine-grained modelling approach is necessary in order to identify the features that play a significant role in the B level. Our proposal is to model the B level according to a set of linguistic metrics. A dataset of systemic and structural complexity metrics is extracted from a Russian L1 corpus of English. The REALEC corpus includes essays written as part of the Russian end-of-course English examination. In order to align them with CEFR levels, we apply two automatic CEFR-grading tools (Duolingo and Textinspector (Bax, 2012)). After analysing the distribution of REALEC grades according to the B1 and B2 levels, we apply the split to a subset of 1,171 essays from the REALEC corpus. The objective is to analyse how much of the B1 vs. B2 classification assigned by the tools can be captured by two families of metrics created with two tools. - Inspector (Vinogradova, 2017) is an online application based on the results obtained in the corpus research across examination texts in REALEC. The tool computes 59 features that were found to be significant for a good grade. The metrics cover four levels of the text, namely, its lexis, syntax, morphology and discourse organisation. - The AIDALLA pipeline (Sousa et al., 2020) is made up of a set of NLP tools which compute lexical (Lu, 2012), syntactic (Lu, 2010), cohesive (Crossley et al., 2016) and semantic (Crossley et al., 2011) features. It computes 768 metrics in total. We compare the performance of the two tools. We discuss the possibility of predicting learner levels on the basis of each pipeline’s feature set. Several methods--random forests, elastic net and K-Nearest Neighbour--are used to identify discriminatory metrics. Results include the comparison of the confusion matrices related to each pipeline as well as the identification of significant predictors of proficiency.
EuroCall2021.pdf (613.65 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03701425 , version 1 (22-06-2022)

Identifiants

  • HAL Id : hal-03701425 , version 1

Citer

Nicolas Ballier, Anton Buzanov, Vinogradova Olga, Thomas Gaillat. A Cross-platform Investigation of Complexity for Russian Learners of English. EUROCALL2021, Sorbonne université, Aug 2021, Paris, France. ⟨hal-03701425⟩
53 Consultations
28 Téléchargements

Partager

More