The speaking rubric redesigned:  Moving away from a dichotomous design

Suzanne Springer, Susan Jackson et Sara Djamàa

École de langues - Université du Québec à Montréal

Questions from the audience in English and French are welcome.

Zoom 859 3601 4296

Returning to the theme of the second edition of MeLT on second language assessment, this presentation reports on the design and development of a revised speaking rubric for use in an English placement test.  Rubrics are central to giving meaning to a test score, and in turn, any claims test developers wish to make about a test taker’s ability, or proficiency claim (Kane et al., 2017; Knoch and Chapelle, 2018).  Heavily influenced by Knoch et al’s 2021 proposed framework for considering several sources in the development of a rater-mediated speaking scale, the rubric that will be presented is the result of contributions from test-external sources, such as expert intuition, course syllabi, and existing scales as well as test-internal sources such as performance samples, rater feedback, and the assessment task itself (p. 13).  Beginning with a self-reflection on what the raters on the panel pay attention to when scoring (Pollitt and Murray, 1993), the scale developers worked together as the merits of the various sources were deliberated and incorporated.   The resulting rubric was then used in the evaluation of 100 speaking samples spanning a range of proficiency levels. This last step allowed for modifications to further refine the rubric.  As test developers are increasingly called upon to defend their proficiency measures and the constructs that underpin them, this presentation provides a practical guide to including multiple sources in rubric development.  Such a process leads to an enriched, more informative rubric that is consistent with a broader commitment to create language assessment tools that are valid, fair, and transparent. 

Références:

Kane, M. T., Clauser, B. E., & Kane, J. (2017). A validation framework for credentialing tests. In C. W. Buckendahl & S. Davis-Becker (Eds.), Testing in the professions: Credentialing policies and practice (pp. 20–41). Routledge.

Knoch, U., & Chapelle, C. A. (2018). Validation of rating processes within an argument-based framework. Language Testing, 35(4), 477–499. 

Knoch, U., Deygers, B., & Khamboonruang, A. (2021). Revisiting rating scale development for rater-mediated language performance assessments: Modelling construct and contextual choices made by scale developers. Language Testing,  38(4), 602-626.

Pollitt, A., & Murray, N.  (1996). What raters really pay attention to. Performance Testing, Cognition and Assessment.  74-91.