Assessing the validity of new structural complexity measures as features of proficiency in L2 English
Résumé
Assessing the validity of new structural complexity metrics as features of L2 proficiency The study of the development of grammatical complexity in writings has been approached with the use of holistic complexity metrics (Bulté & Housen, 2012; Norris & Ortega, 2009) and the use of syntagmatic forms mapped to specific functions (Biber et al., 2011, 2020, 2023). In both cases, lexico-grammatical patterns are counted along the syntagmatic axis to compute proportions or ratios, and analyzed in terms of correlations with proficiency/developmental stages. For all their benefits, these studies have left paradigmatic production out of the equation. Few studies have focused on analysing the internal structure of a form-function mapping despite evidence of paradigmatic instability within mappings (Bresnan et al., 2007; Gaillat et al., 2021). Based on the construct of microsystems (Gentilhomme, 1980; Py, 1980), we operationalise the measurement of the probability of occurrence of a form vs its paradigmatic competitors within mappings. For instance, we compute the probability of occurrence of proform THIS vs THAT or IT. To do so, after pre-processing the EFCAMDAT corpus (Shatz, 2020) with UDPipe (Straka et al., 2016), we extract the forms of a microsystem with GrewMatch (Guillaume, 2021) and train a model (multinomial logistic regression with Nnet library in R) on a randomized training subset (80%). We apply the same extraction strategy and the model prediction on the test set. We use ordinal logistic regression to evaluate associations between probabilities and CEFR-based proficiency. Premiminary results for the proform microsystem show that the odds of the median of the forms’ probabilities per text are associated with proficiency (Kruskal-Wallis rank sum test across CEFR: p < .001). The ordinal regression indicates that THAT and THIS probabilities are strongly associated with higher levels, whilst IT probability is associated with low levels. Other candidates for microsystems include groups of determiners, relative pronouns and multiword structures. These new metrics provide measures of paradigmatic dimension in the assessment of writing.