DOI: https://doi.org/10.14710/kiryoku.v10i2.710-723

A Comparative Study of Human and AI Assessment in Japanese Sakubun Writing

1Universitas Pendidikan Indonesia, Indonesia

2Nagasaki University of Foreign Studies, Nagasaki, Japan, Japan

Open Access Copyright (c) 2026 by authors under http://creativecommons.org/licenses/by-sa/4.0.

Citation Format:
Abstract

This study compares the assessment of Japanese sakubun by a native Japanese teacher and two large language model based AI tools, Gemini AI (3.1) and ChatGPT (5.5), and examines the extent to which AI generated scores align with human assessment. The data comprised eight sakubun texts on the theme of meiwaku (inconvenience) written by second year students with approximately JLPT N4 level Japanese proficiency. Each sakubun was assessed using the same rubric covering four dimensions: content, organization, expression, and accuracy, as well as an overall score. The scores were descriptively compared across the three evaluators at both the criterion and overall levels. The results showed that the Japanese teacher assigned the highest average overall score of 16.25, followed by Gemini AI with 14.25 and ChatGPT with 13.75. Of the two AI tools, Gemini AI demonstrated closer overall alignment with the teacher’s assessment. Both AI systems also exhibited a score compression tendency. Higher quality sakubuns were generally evaluated lower than by the teacher, whereas some lower quality sakubuns received relatively higher scores. The most substantial discrepancies were observed in content and expression, indicating that AI assessment was less sensitive to idea development, contextual appropriateness, naturalness of expression, and subtle linguistic nuances. These findings suggest that although Gemini AI and ChatGPT can provide useful supplementary assessment, human evaluator judgment remains essential, particularly for evaluating qualitative and context sensitive aspects of Japanese writing. 

Fulltext View|Download Email colleagues
Keywords: Japanese writing assessment; artificial intelligence; automated writing assessment; human–AI agreement; Japanese language education

Article Metrics:

Article Info
Section: Articles
Language : EN
Most cited articles
Verba Kompon Hajimeru, Dasu, Dan Kakeru Sebagai Penanda Aspek Inkoatif Bahasa Jepang More cited articles
  1. Bouziane, K., & Bouziane, A. (2024). AI versus human effectiveness in essay evaluation. Discover Education, 3(1), 201. https://doi.org/10.1007/s44217-024-00320-6
  2. Doi, K., Sudoh, K., & Nakamura, S. (2024). Automated Essay Scoring Using Grammatical Variety and Errors with Multi-Task Learning and Item Response Theory. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 316–329
  3. Hirao, R., Arai, M., Shimanaka, H., Katsumata, S., & Komachi, M. (2020). Automated essay scoring system for nonnative Japanese learners. Proceedings of the Twelfth Language Resources and Evaluation Conference, 1250–1257
  4. Johnson, M., & Zhang, M. (2024). Examining the responsible use of zero-shot AI approaches to scoring essays. Scientific Reports, 14(1), 30064. https://doi.org/10.1038/s41598-024-79208-2
  5. Lazendic, G. (2023). The validity of automated essay scoring. The Laboratory for Experimental Psychology Univerzitet u Beogradu
  6. Lee, J.-H., & Hasebe, Y. (2020). Quantitative Analysis of JFL Learners’ Writing Abilities and the Development of a Computational System to Estimate Writing Proficiency. Learner Corpus Studies in Asia and the World, 5, 105–120
  7. Li, S., & Ng, V. (2024). Automated Essay Scoring: A Reflection on the State of the Art. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 17876–17888. https://doi.org/10.18653/v1/2024.emnlp-main.991
  8. Li, W., & Liu, H. (2024). Applying large language models for automated essay scoring for non-native Japanese. Humanities and Social Sciences Communications, 11(1), 723. https://doi.org/10.1057/s41599-024-03209-9
  9. Mansour, W., Albatarni, S., Eltanbouly, S., & Elsayed, T. (2024). Can Large Language Models Automatically Score Proficiency of Written Essays? Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation , 2777–2786
  10. Oktaviani, A. N., Alief, M. Z., Santiar, L., Purnamasari, P. D., & Ratna, A. A. P. (2021). Automatic Essay Grading System for Japanese Language Exam using CNN-LSTM. 2021 17th International Conference on Quality in Research (QIR): International Symposium on Electrical and Computer Engineering, 164–169. https://doi.org/10.1109/QIR54354.2021.9716165
  11. Pack, A., Barrett, A., & Escalante, J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6, 100234. https://doi.org/10.1016/j.caeai.2024.100234
  12. Prakoso, A. A., Salsabila, K., Yahya Maulana, A., & Erza, E. K. (2025). Hubungan penggunaan teknologi artificial intelligence gemini dalam mendukung kebutuhan informasi mahasiswa. Pustaka Karya : Jurnal Ilmiah Ilmu Perpustakaan Dan Informasi, 13(2), 359–372. https://doi.org/10.18592/pk.v13i2.18418
  13. Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527. https://doi.org/10.1007/s10462-021-10068-2
  14. Saricaoglu, A., & Bilki, Z. (2025). The capacity of ChatGPT-4 for L2 writing assessment: A closer look at accuracy, specificity, and relevance. Annual Review of Applied Linguistics, 45, 253–273. https://doi.org/10.1017/S0267190525100160
  15. Tajima, M. (2010). 日本語の文章に対する分析的評価の信頼性に関しての検証 [Verification of the reliability of analytic assessment for Japanese writing]. Chuo Gakuin University Journal of Humanities and Natural Sciences, 31, 85–104. https://cgu.repo.nii.ac.jp/records/1469
  16. Utari, R., Sukmara, R., & Septiany, N. (2024). Implementation of Project-Based Learning In Improving The Creativity of Writing Japanese Essays in The Sakubun Course. KIRYOKU, 8(1), 139–146. https://doi.org/10.14710/kiryoku.v8i1.139-146
  17. Wahani, Y. J. (2025). Task-Based Language Teaching (TBLT) Learning Model using Worksheets to improve Japanese writing skills. KIRYOKU, 9(1), 40–50. https://doi.org/10.14710/kiryoku.v9i1.40-50
  18. Yoon, S., Miszoglad, E., & Pierce, L. R. (2023). Evaluation of ChatGPT feedback on ELL writers’ coherence and cohesion. ArXiv
  19. Zahira, N. A., Nurhafidzoh, S. U., Annur, M. K., & Saputra, D. Y. (2025). Pengaruh Penggunaan Artificial Intelligence Terhadap Kemampuan Menulis Akademik Mahasiswa. Sindoro : Cendikia Pendidikan, 18(2), 2661–2670
  20. Zhang, T., Erlam, R., & de Magalhães, M. B. (2025). Exploring the dual impact of AI in post-entry language assessment: Potentials and pitfalls. Annual Review of Applied Linguistics, 45, 274–293. https://doi.org/10.1017/S0267190525000030

Last update:

No citation recorded.

Last update: 2026-10-17 13:28:44

No citation recorded.