Connecting Criterion scores and Classroom Grading Contexts: A Systemic Functional Linguistic Model for Teaching and Assessing Causal Language
Issue: Vol 33 No. 1 (2016) Automated Writing Evaluation
Journal: CALICO Journal
Subject Areas:
Abstract:
This study utilized theory proposed by Mohan, Slater, Luo, and Jaipal (2002) regarding what they refer to as the Developmental Path of Cause to examine issues of AWE score use in classroom contexts. Utilization of this model enabled this study to investigate the accuracy of the AWE scores by comparing them to ratings based on teachers’ intuition as well as to scores generated based on existing rubrics. The qualitative data collected from focus group interviews of three experienced teachers’ justifications for their intuitive evaluations of essays suggested that the Developmental Path of Cause helped teachers articulate their intuitions, identifying the core features of the model. The quantitative results showed that the grades provided by raters trained to use the Developmental Path of Cause tended to support Criterion scores more strongly than did instructor grades. The findings from this study suggest that AWE scores from Criterion not only closely correlated with teachers’ intuitions and with raters trained to use the Developmental Path of Cause, but that the use of the Developmental Path of Cause for teaching may support the use of AWE systems in the classroom context, and would help students focus on the core of a cause-effect essay: appropriateness and sophistication of causal language.
Author: Hong Ma, Tammy Slater
References :
American Council on the Teaching of Foreign Languages. (2002). Program standards for the preparation of foreign language teachers (Initial level- undergraduate & graduate) (For K-12 and secondary certification programs). Retrieved from http://www.actfl.org/sites/default/files/pdfs/public/ACTFLNCATEStandardsRevised713.pdf.
Attali, Y., & Burstein, J. (2005). Automated essay scoring with e-rater version 2.0 (ETS RR-04-45). Princeton, NJ: Educational Testing Service.
Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a generic approach in automated essay scoring. The Journal of Technology, Learning and Assessment, 10 (3), 1–17.
Bachman, L. (2004). Statistical Analysis for Language Assessment. New York: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511667350
Burstein, J., Chodorow, M., & Leacock, C. (2003a). Criterion online essay evaluation: An application for automated evaluation of student essays. AI Magazine, 25 (3), 27–35. http://dx.doi.org/10.1609/aimag.v25i3.1774
Burstein, J., Marcu, D., & Knight, K. (2003b). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems: Special Issue on Natural Language Processing, 18 (1), 32–39. http://dx.doi.org/10.1109/MIS.2003.1179191
Chen, C.-F.E., & Cheng, W.-Y.E. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12 (2), 94–112.
Cheville, J. (2004). Automated Scoring Technologies and the Rising Influence of Error. English Journal, 93 (4), 47–52. http://dx.doi.org/10.2307/4128980
Conference on College Composition and Communication (2006). Writing assessment: A position statement. Retrieved July 20, 2007, from http://www.ncte.org/cccc/resources/positions/123784.htm.
Deane, P. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18, 7–24. http://dx.doi.org/10.1016/j.asw.2012.10.002
Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5 (1). Retrieved from http://www.jtla.org
Ebyary, K., & Windeatt, S. (2010). The impact of computer-based feedback on students’ written work, International Journal of English Studies, 10 (2), 121–142.
Ericsson, P. F. (2006). The meaning of meaning: Is a paragraph more than an equation? In P. F. Ericsson & R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences, 28–37. Logan: Utah State University Press.
Green, A. (2007). IELTS washback in context: Preparation for academic writing in higher education. Cambridge, UK: Cambridge University Press.
Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8 (6), 4–44.
Halliday, M. A. K. (1998). Things and relations: Regrammaticising experience as technical knowledge. In J.R. Martin & R. Veel (Eds), Reading science: Critical and functional perspectives on discourses of science, 185–235. New York: Routledge.
Halliday, M. A. K., & Martin, J. R. (1993). Writing Science: Literacy and Discursive Power. Washington DC: The Falmer Press.
Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The Role of Automated Writing Evaluation Holistic Scores in the ESL Classroom. SYSTEM Journal, 44, 66–78. http://dx.doi.org/10.1016/j.system.2014.02.007
Klenbaum, D. G., Kupper, L. L., & Muller, K. E. (1988). Applied regression analysis and other multivariable methods. Boston: PWS-KENT Publishing Company.
Low, M. (2010). Teachers and texts: Judging what English language learners know from what they say. In A. Paran & L. Sercu (Eds), Testing the untestable in language education (pp. 241–255). Bristol, UK: Multilingual Matters.
Messick, S. (1996). Validity and washback in language testing. Princeton, NJ: Education Testing Services.
Mohan, B., & Slater, T. (2004). The evaluation of causal discourse and language as a resource for meaning. In J. A. Foley. (Ed.), Language, education & discourse: Functional approaches, 255–269. London: Continuum.
Mohan, B., Slater, T., Luo, L., & Jaipal, K. (2002). Developmental lexicogrammar of causal explanations in science. Paper presented at the International Systemic Functional Linguistics Congress (ISFC29), Liverpool, UK.
Saldana, J. (2009). The coding manual for qualitative researchers. Washington, DC: SAGE.
Slater, T. (1998). Evaluating causal discourse in academic writing. MA thesis. University of British Columbia.
Slater, T. (2004). The discourse of causal explanations in school science. PhD thesis, University of British Columbia.
Slater, T., & Mohan, B. (2010). Towards systematic and sustained formative assessment of causal explanations in oral interactions. In A. Paran & L. Sercu (Eds), Testing the untestable in language education, 256–269. Bristol, UK: Multilingual Matters.
Wang, J., & Brown, M. S. (2007). Automated essay scoring versus human scoring: a comparative study. Journal of Technology, Learning, and Assessment, 6 (2).
Ziegler, W.W. (2007). Computerized Writing Assessment: Community College Faculty Find Reasons to say ‘Not Yet’. In P. F. Ericsson & R. Haswell (Eds.), Machine Scoring of Human Essays: Truth and Consequences, 138–153. Logan, Utah: Utah State University Press.