Evaluating the consistency of automated CEFR analyzers: a study of English language text classification

Piyathat Siripol, Seongha Rhee, Suthathip Thirakunkovit, Aphiwit Liang-Itsara

Abstract


With the increasing use of web-based tools for text analysis, there is a growing reliance on automated systems to assess text difficulty and classify texts to the Common European Framework of Reference for Languages (CEFR). However, inconsistencies in these tools’ outputs could undermine their effectiveness for language learners and researchers. This study investigates the consistency of five widely used automated CEFR analyzer tools, including ChatGPT, by analyzing 20 English descriptive texts at CEFR levels B1 and B2. A quantitative approach was employed to compare the CEFR classifications generated by these tools. The results reveal significant inconsistencies across the tools, raising concerns about the reliability of automated CEFR alignment. Additionally, the content and genre of texts appeared to influence the CEFR classification, suggesting that certain factors beyond the tools’ algorithms may affect their accuracy. These findings have important implications for language educators, curriculum designers, and researchers who rely on automated CEFR tools for text selection, grading, and analysis. The study highlights the limitations of automated CEFR classification systems and calls for a more qualitative approach to text difficulty alignment analysis. Future research recommendation is discussed and call for more focus on refining these tools and exploring additional factors that may impact their effectiveness in text difficulty measurement and CEFR alignment.

Keywords


Analyzer tools; Automated text analysis; CEFR; Text difficulty classification; Web based analysis

Full Text:

PDF


DOI: http://doi.org/10.11591/ijere.v14i4.33528

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Institute of Advanced Engineering and Science

International Journal of Evaluation and Research in Education (IJERE)
p-ISSN: 2252-8822e-ISSN: 2620-5440
The journal is published by Institute of Advanced Engineering and Science (IAES).

View IJERE Stats

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.