Development and Validation of a Web-based L2 Pragmatic Speaking Test in the University Context

Chen, Shi (2021) Development and Validation of a Web-based L2 Pragmatic Speaking Test in the University Context. Doctoral thesis, Northern Arizona University.

Text Chen_2021_development_validation_web-based_l2_pragmatic_speaking_test_.pdf - Published Version Restricted to Repository staff only Download (4MB) | Request a copy

Abstract

Pragmatic competence has been increasingly recognized as a component of academic language proficiency assessment (Youn, 2018). However, the practicality of administering L2 pragmatic competence assessments involving interlocutors remains problematic (e.g., high labor cost) (Ikeda, 2017). The pandemic makes it even more difficult to administer role-play or paired speaking tasks in person. Therefore, there is a strong need for such pragmatic speaking tests to be administered online. Another issue of L2 pragmatic competence assessment is construct under-representation. The constructs of performance-based assessments and some English proficiency tests (e.g., IELTS speaking) also require additional investigation (Roever, 2011) because the tests are not completely measuring what testing agencies intend to measure. The primary purpose of this study is to present the test development process and validity evidence for the Second Language Pragmatic Test (L2PT). This online test was developed to serve as a diagnostic test to evaluate pragmatic competence in spoken interaction for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners. The target test takers were either studying or wanted to study at a North American University. The web-based L2PT was developed using Python and JavaScript, and was hosted using Amazon Web Services (AWS). The test has three sections which includes 25 test items and takes approximately an hour to complete. The development of the L2PT was guided by the Discursive Approach to L2 Pragmatics (Kasper, 2006) framework, and informed by politeness theory (Levinson, 1987). The L2PT intends to measure constructs including task completion, language use, turn-taking, and interactional fluency. The language use situations of the L2PT were chosen based on previous needs analysis (Youn, 2018), corpus study (Conrad et al., 2006), and expert opinions. The rating criteria development was based on a data-driven approach (Youn & Chen, 2021). The L2PT and the rating criteria went through multiple rounds of piloting and revisions. The test validation process utilizes the argument-based validity framework (Kane, 2013; Knoch & Chapelle, 2017; Chapelle et al., 2008). To provide validity evidence of the test, 112 EFL and ESL learners and eight raters were recruited. The rating was designed using an anchored rating design in which raters scored 30% of test takers’ performance and the remaining test takers were scored by at least two raters. The validation process involves a Sequential Mixed Methods approach. For the quantitative analysis, the raters’ scores were analyzed using the Multi-Facet Rasch Measurement (MFRM) as implemented in the FACETS (Linacre, 2014) software. Four facets were included in the analysis: test takers, raters, rating criteria, and tasks. In terms of qualitative analysis, post-rating interviews were conducted from the raters to shed light on the rating experience, rating criteria, and test design. The raters’ insights and comments were summarized using content analysis. The discussion of the quantitative and qualitative results is presented following Kane’s (2013) validity framework. For the domain inference, the results show that the development process of L2PT followed a good test development practice. For the evaluation inference, although the 4-point scale and the test items were relatively easy for the test takers, the rating criteria for the L2 pragmatic speaking test can function as intended in an EAP context in relation to the rating process, and the test administration process was appropriate. As for the generalization inference, there is some evidence to show that test takers are expected to receive scores that can measure their true ability. The findings reveal the following: (1) The possibility of administering L2PT online, (2) Sufficient evidence to support the domain definition, evaluation inference, and generalization inference, (3) Design process for future pragmatic speaking tests. This dissertation exemplifies a positive step toward developing the pragmatic speaking test that can be administered online. The conclusion also provides methodological, research and pedagogical implications for using and implementing the L2PT.

Item Type:	Thesis (Doctoral)
Publisher’s Statement:	© Copyright is held by the author. Digital access to this material is made possible by the Cline Library, Northern Arizona University. Further transmission, reproduction or presentation of protected items is prohibited except with permission of the author.
Keywords:	language testing and assessment ; online English test ; pragmatic competence in spoken interaction
Subjects:	P Language and Literature > P Philology. Linguistics
NAU Depositing Author Academic Status:	Student
Department/Unit:	Graduate College > Theses and Dissertations College of Arts and Letters > English
Date Deposited:	07 Feb 2022 17:03
Last Modified:	07 Feb 2022 17:03
URI:	https://openknowledge.nau.edu/id/eprint/5669

Actions (login required)

IR Staff Record View

Downloads