Three stakeholders’ reflections on language assessment literacy

Abstract

This reflective action research documents the mutual learning experiences of three education stakeholders as they construct their language assessment literacy (LAL). A teacher educator, an inservice teacher, and a preservice teacher were the three stakeholders who took part in a project on language assessment development for ESOL students at a public school in the United States. In this project, with the help of inservice ESOL teachers, preservice ESOL teachers designed and implemented listening and writing assessments for local middle school ESOL students. During and after the project, one of the participating inservice teachers, one preservice teacher, and the teacher educator voluntarily participated in extended reflective practices via email communications, on-site visits, and pre- and post-project discussions. All stakeholders perceived the collaboration as a mutual learning experience to enhance their LAL while sharing their respective expertise.

Introduction

The present reflective action research describes the mutual learning experience of developing language assessment literacy (LAL) among a teacher educator (the first author, Jiyoon), an inservice ESOL teacher (the second author, Natalia), and a preservice ESOL teacher (the third author, Debby). The three stakeholders participated in a semester-long language assessment development project for preservice ESOL teachers. In the project, 16 preservice teachers had a chance to work with two inservice ESOL teachers and their ESOL students in neighbouring schools. Among the pre and inservice teachers, Debby and Natalia volunteered to extend their participation in the project. With Jiyoon, they communicated extensively via emails, on-site visits, and pre- and post-project discussions. Once the project was completed, the three stakeholders had multiple reflection sessions about their experience during the project and analyzed their experiences of enhancing their LAL. While previous studies which investigate teachers’ LAL development are often from top-down and researcher-directed perspectives, this study presents a teacher training model that connects all stakeholders in teacher education as equal partners.

Literature review

The concept of LAL was first introduced in Brindley (2001). In designing language teacher training modules, Brindley presented the importance of teachers’ competences in language assessment. He particularly emphasized language teachers’ understanding of social roles and consequences of language assessment in his training module. Along with Brindley’s arguments, the components of LAL were identified using language assessment textbook analyses, pre and inservice teacher surveys, and interviews. One of the widely accepted LAL definitions refers to stakeholders’ (1) theoretical knowledge of language assessment theories; (2) practical skills to design, implement, and interpret language assessment and its results; and (3) understanding of the cultural, ethical, pedagogical, political, and social consequences of language assessment uses (Davies 2008; Inbar-Lourie 2008; Fulcher 2012; Lee 2019).

While the concept of LAL keeps evolving (Berry, Sheehan, and Munro 2018), what has been constant is teachers’ high LAL needs. Their high LAL needs reflect their extensive engagement in assessment practices including both formative and summative assessments. They develop and use classroom assessment, administer high stakes testing, and interpret testing results for students and their caregivers. The frequent uses of language assessment in schools and society demand high levels of teachers’ LAL among all other stakeholders.

Reflecting the urgency of enhanced teachers’ LAL, active research and conversations about teachers’ LAL have been conducted in both fields of teacher education and language assessment (Lee, Butler, and Peng 2021; Vogt and Tsagari 2014). Research has revealed that both pre and inservice teachers do not receive sufficient training on language assessment (Fulcher 2012; Vogt and Tsagari 2014). In contrast to their lack of training, in daily teaching teachers spend more than one third of their time on assessment-related activities (Stiggins 2007). A survey reported that the pressure from considerable assessment activities is one of the critical reasons why early career teachers leave their profession (Owens 2015). On a more optimistic note, however, research has also reported a positive effect of focused teacher training to enhance pre and inservice teachers’ LAL to a certain extent (Koh et al. 2018; Lee et al. 2021).

With time constraints in preservice teacher education, teacher educators often encounter challenges in introducing the components of LAL to their full extent to preservice teachers. The efforts to enhance preservice teachers’ LAL have been mostly top-down, knowledge transmission approaches that are from experts (i.e., teacher educators or assessment specialists) to novices (i.e., preservice teachers) with experts presenting assessment theories, techniques or skills, and assessment consequences to novices. In this approach, it is challenging to accommodate and reflect the specific aspects of individual teachers’ teaching contexts. As such, top-down approaches have their limitations.

Documenting the interaction between teachers and language testing professionals (i.e., workshop facilitators), Baker and Riches (2018) presented the collaboration between the stakeholders as an alternative approach. They reported that teachers developed a deeper understanding of the connections between teaching and assessment and their roles as supporters for their students’ successful experience in assessment. Meanwhile, language testing professionals gained understanding of the specifics of the context and target learners. Based on their findings, Yan and Fan (2021) suggested an ‘apprenticeship-based, experience-mediated model of LAL development’ (238). They emphasized the collaboration and reflection in assessment activities for enhancing stakeholders’ LAL. This model also acknowledges that every stakeholder has some level of LAL.

To take one step further, this study adopted reflective action research to closely examine what each stakeholder (i.e., a preservice, an inservice, and a teacher educator) contributed to teacher education related to LAL and what they have learned through the collaboration.

Methods

Setting

The three authors participated in a semester-long language assessment course for ESOL preservice teachers in three respective roles. In this course, 16 preservice teachers were divided into four groups and designed listening and writing assessments for ESOL students at two local public schools. The teacher educator, Jiyoon, recruited two inservice ESOL teachers who could provide access to their ESOL students to preservice teachers and share their experiences of implementing the newly designed assessment among their students. The purpose of this collaboration was to help preservice teachers design language assessments for real students. Through the collaboration, they could also have opportunities to apply theoretical knowledge they have gained through the course work and practice their assessment skills. The language assessment development project was composed of four stages, which were planning, designing, implementing, and reflecting. The preservice ESOL teachers conducted ESOL students’ needs analysis (test-taker report), designed listening and writing assessments, analysed the assessment results, and reflected on the processes. The inservice ESOL teachers implemented the assessments among their ESOL students and shared their feedback and their students’ comments with the preservice teachers and the teacher educator. The preservice teachers were required to submit multiple reflection pieces at varied stages of the project. It was not required, but the teacher educator offered opportunities for the preservice teachers to join her in visiting inservice teachers before and after the project.

Participants

Once the semester was over, Jiyoon invited a preservice teacher, Debby and an inservice teacher, Natalia to further reflect on their collaboration. During the project, Debby and Natalia were exceptionally dedicated to their responsibilities. Debby was one of two preservice teachers who visited inservice teachers prior to and after the project. Natalia has been collaborating with Jiyoon on a preservice teachers’ language assessment development project since 2015. Natalia provided extensive and critical feedback to preservice teachers’ language assessment. Her feedback centered on her observation of her own ESOL students’ reaction and performance on the preservice teachers’ language assessment. Her feedback was invaluable as preservice teachers were able to experience how their assessment played out in a real classroom. Natalia was educated in Russia and in the United States and has been teaching ESOL students at a public elementary middle school in a city in the Mid-Atlantic region of the United States since 2002. She has served as a mentor teacher during teaching internships and has been involved in providing professional development to ESOL teachers in the district.

Debby has taken multiple courses with Jiyoon and was one of the most involved preservice teachers. Debby’s reflections presented her critical thinking on language education in general and language assessment in particular. After completing her master’s degree in TESOL, Debby now teaches at a public school in the same city where Natalia teaches. She is a passionate career changer and has dedicated herself to supporting minority communities in the local area and abroad for an extended period of time. Jiyoon completed her primary, secondary, and college education in South Korea and received a master’s in TESOL and a Ph.D. in Educational Linguistics in the USA. She has been teaching language assessment to preservice teachers since 2012, and her research centers on preservice teachers’ and ESOL learners’ LAL.

Data collection and analysis

This study adopts a reflective action research method where we emphasized extended and collaborative reflections. Reflective action research helps researchers study concerning issues systematically through critical reflections (Corey 1953). This method is well-received by practitioners who perceive problems in their own practice including teaching and attempt to take action to improve their situation. Therefore, the goals of action research are to evaluate, improve, make decisions, and take action. In that way, it is considered ‘a small-scale intervention’ (Cohen and Manion 1994: 186). Action research becomes more rigorous and yields valid results when researchers take a collaborative approach to data collection as well as reflection. The importance of collaboration is to reflect everyone’s perspectives to understand and take action. The collaboration in action research enhances participants’ understanding of each other’s stances and helps them take appropriate actions.

This study’s data came from two sources: the first set of data consists of recorded conversations at the initial and debrief meetings of the language assessment development project. The recorded conversations were transcribed by an undergraduate research assistant and cross-checked by the first author. The second set of data consists of three authors’ written reflections, which includes the three researchers’ expectations before the project and reflections once the project was over.

The three authors adopted an inductive thematic analysis approach (Nowell et al. 2017) in analysing the data. Following the approach, we immersed ourselves in the data independently first. We chose phrases, comments, and statements that best reflected our collaborative learning experience during the project. Then, we had three online meetings to talk about what descriptions we had chosen and to explain why they had caught our attention. We discussed our individual strengths and contributions to collaboration and mutual learning experience, which were evidenced in our data. Then, we identified the quotes’ main themes. The first author took the lead in the discussions and data analysis as she had a research background; however, all the authors contributed collectively to identifying themes and to data analysis.

Results and discussions

The data revealed the benefits of collaboration across all three stakeholders. It was also clear that every stakeholder had strengths and contributed to each other’s LAL enhancement. Jiyoon’s expertise in language assessment theories, Natalia’s insight of her own students and education system, and Debby’s critical feedback of her own education experience gave them chances to reflect on their LAL. It was important to note that the contribution was not unidirectional from experts to novices; all participants had LAL as a result of their past language assessment experience. Each had respectively extensive knowledge of language assessment in different contexts (e.g., South Korea, Russia, and the United States) and in different roles (e.g., as a learner, a teacher, or a community leader).

The three most prominent themes that came from the three authors’ reflections included (1) understanding of language assessment theories through the collaboration; (2) learner-centered approaches to LAL; and (3) the importance of connecting the real classroom experience to teacher education.

Theoretical knowledge

One of the noticeable learning experiences that all stakeholders had was a deepening of their understanding of language assessment theories. In LAL models, stakeholders’ understanding of language assessment theories is one of the most important aspects.

Jiyoon, who is a trained language assessment specialist and teacher educator, brought her expertise in language assessment theories. She presented theoretical concepts including validity, reliability, interactiveness, authenticity, impact, and practicality (Bachman and Palmer 2002) to her preservice teachers. However, she soon realized that the jargon would not be clearly applicable in the language assessment development project and real classroom setting. Hence, she avoided jargon and instead used examples to explain those concepts to preservice teachers. By making potentially esoteric language assessment theories clear for her audience, Jiyoon was able to enhance her own grasp of language assessment theories.

It turned out that Natalia also found the theoretical accounts of language assessment helpful for developing her LAL.

Natalia stated,

Through our discussions, I understood that one of the focuses of the course was the Framework of Usefulness of assessments, in which the preservice teachers examined construct validity, reliability, authenticity, practicality, interactiveness, and impact of the assessments.

Natalia conveyed that her teacher education program did not cover the language assessment theories. She expressed her desire to delve deeper into the subject. Natalia mentioned that she was able to increase her knowledge of language assessment theories from the language assessment development project. She was also able to contribute her lived knowledge as an experienced ESOL teacher to the project. Natalia provided critical information about her students’ proficiency data (e.g., World Class Instructional Design and Assessment (WIDA) scores). But the more important information that only Natalia was able to provide was her insight about her students’ preference and interests in topics. Natalia shared curriculum materials, district policies, her lesson plans, and other additional reading and activity materials she used with her students. Jiyoon encouraged her preservice teachers to use the information to select listening passages or writing topics, which helped the ESOL students relate to the assessment content. The detailed information about Natalia’s teaching became the basis of formative assessment that preservice teachers designed.

Debby, a preservice teacher, found the theoretical knowledge and insider insight provided by Jiyoon and Natalia to be organically connected to each other. She stated, ‘the instructor assigned various examples and readings throughout the semester that were useful in guiding the preservice teacher to create meaningful assessments’.

Debby continued,

The studies that we read/researched in class, although helpful, did not give us (preservice teachers) the total grasp of what we were doing until we were engaged in an actual hands-on opportunity and received students’, teacher educator’s, and inservice teacher’s feedback on our created assessments.

In Debby’s reflection, it was clear that she learned a lot from the language assessment development project. In her assignment that discussed her language assessment experience, Debby stated that she did not have much experience in giving or developing language assessments. Her experience as a learner who went through the U.S. education system and as a tutor who worked with refugees set her up as a key informer in the language assessment development project. Jiyoon found Debby’s critical reflections on her teaching constructive. Jiyoon could see what she needs to clarify or emphasize for preservice teachers.

Learner-centered approaches to LAL

All stakeholders agreed that it was important to start the language assessment development project from the knowledge of learners. The first task of the language assessment development project was a test-taker report. In the report, preservice teachers created survey or interview questions to collect information about the ESOL students for whom they would develop their language assessment.

Natalia mentioned,

The process of creating language assessments that take into consideration students’ language proficiency levels, their background knowledge, and interests is important in providing equitable educational opportunities to the students.

It is critical to note that learner-centered approaches to language assessment not only reflect learners’ knowledge or preference in language assessment but also motivate learners to be part of the whole process of language assessment. Natalia’s understanding of learner-centered approaches was clearly captured in the following quote:

The approach to assessments has changed during the past several years. There is more emphasis on formative assessments and their impact on planning of the next learning cycle, offering student support, and forming small groups. The importance of student choice and voice is also brought to the forefront. I expected the assessments created by the participants to be learning experiences for my students and a chance for them to show content and language skills in a way that takes into consideration their interests and language proficiency.

Working with actual test-takers helped Debby realize the efficacy of her group’s language assessment. She mentioned,

All with good intentions, the assessments that were created tended to be either overzealous for the student groups or way underestimated for the students’ academic language abilities. Having the opportunity to work with actual students opposed to a hypothetical situation gave the preservice teacher an insight to reality, rather than what may appear adequate on paper.

In many teacher education settings, due to limited resources, working with actual learners is not always feasible. However, Debby’s comments on creating assessments for hypothetical learners that seem ‘adequate on paper’ provide critical moments of reflection for teacher educators. Like other instruction materials, unless language assessment is tested out with actual learners, the quality and usefulness of language assessment will be in question.

Jiyoon proposed a LAL framework that placed test-takers in the centre. In the model, she argued that all language assessment should start from stakeholders’ understanding of learners (Lee 2019). By centering learners in the LAL model, the participants raised the following questions:

I was able to see how my knowledge of the students’ language proficiency levels, confidence levels, and scaffolding needs helped me predict the students’ success with some assignments and difficulties with others. It was beneficial to learn more about my students by observing their participation in the assessment exercises. Who was comfortable with technology? Who was frustrated by it? Who needed more time to play the listening prompt several times? Who would need further typing practice? What assessment features did the students enjoy? What was confusing for them and why? What would be good to incorporate in the future assessments? (Natalia in her reflection)

These questions are critical to reconceptualize LAL as a contextual understanding of language assessment. That is, these questions help stakeholders contextualize assessment purposes, types, and uses.

Importance of connecting the real classroom and teacher education

The collaboration also highlighted the importance of early field experience. In general, preservice teachers will encounter real classroom settings and students in their final year of teacher education. They are placed in public schools for their field placements as part of U.S. teacher education. In the state where this study was conducted, preservice ESOL teachers begin their first placement in their second-to-last semester (Phase 1). In Phase 1, preservice ESOL teachers spend a total of 20 days at a public school. During this phase, they observe inservice teachers but are not obligated to teach or work with ESOL students. In Phase 2, preservice ESOL teachers are required to be at their assigned schools for 80 days. In this phase, they design their own lesson plans and teach, initially alongside inservice teachers, and later on their own.

The benefits of connecting the real classroom and teacher education were well captured in Debby’s reflection. Debby stated,

Both of these collaborations were extremely beneficial: an opportunity to grasp what and how the in-service teacher was actually teaching, and the background knowledge she had on her students that could be made privy to the preservice teachers.

Debby’s statement is what preservice teachers usually experience in Phase 2 of their rotation. In particular, observing the consequences of language assessment and impact of their assessment on test-takers is not always obtainable during the short period of field placement. Through this collaboration, the preservice teachers had a chance to examine how real ESOL learners performed on their assessments and how the feedback that they provided was used by the ESOL learners. Debby stated,

In the end, the inservice teacher was able to communicate on what worked with the given assessments, how the students initially responded when taking the assessments, and her suggestions on improving student expectations.

The invaluable experience that Debby had in relation to working with real test-takers assisted Debby in enhancing her LAL.

Natalia also found connecting teacher education with real classroom experience strategically important. Natalia considered that the collaboration helped her enhance her leadership as an outstanding teacher in the district. Natalia mentioned,

I also expected to be helpful in the district’s effort of new teacher recruitment. I believe the preservice teachers have the great potential of bringing their talent, energy, leadership, new methodology and passion for teaching into the schools. Showing them that there are opportunities for professional development, teacher leadership, and advocacy, as well as application of their theoretical knowledge in the classroom can be a factor in choosing a school district to apply to for their future employment.

Through the language assessment development project and her collaboration with the teacher educator and her preservice teachers, Natalia saw potential for recruiting new teachers and situating them in the classroom promptly.

As an educator of the next generation of ESOL teachers, who will highly impact their students’ lives, Jiyoon felt the sense of responsibility. Connecting the real classroom to teacher education helped Jiyoon contextualize her understanding of current ESOL instruction and learners. She could ensure that her instruction could be authentic, and her students could apply LAL they gain from her course as they start their teaching.

Conclusions and future directions

While the collaboration was mostly successful, some preservice teachers found the collaboration overwhelming. While it was offered to all the preservice teachers in class, only two preservice teachers (Debby and one more preservice teacher) volunteered to visit the collaborating teachers’ class and have meetings at the beginning and end of the project. Moreover, some stated that the step-by-step process to design an assessment that reflects both Jiyoon’s and the inservice teachers’ feedback was challenging.

However, as Debby mentioned, experiencing the collaboration during her teacher education helped her and the other preservice teachers keep their feet on the ground. The preservice teachers learned the complex nature of second language assessment development before they start actual teaching. The collaboration certainly gave an extensive chance for the preservice teachers to develop their LAL.

As indicated in Natalia’s reflection, even experienced inservice teachers seek opportunities to enhance their LAL.

Natalia stated,

I expected to increase my own Language Assessment Literacy through this collaboration. The feedback for my National Board for Professional Teaching Standards (NBPTS) portfolio entry on assessment indicated a larger area of needed growth compared to three other entries I submitted. I decided to focus my professional development efforts on learning more about creating effective assessments for ELLs, and this work fit perfectly in this plan.

Through an integrated process of language assessment development that connected teacher educators, preservice teachers, and inservice teachers, this study addressed the importance of contextualized teacher education, especially on stakeholders’ LAL development.

Furthermore, although students’ voices were not rigorously documented in this study, it was evident that students were thrilled to receive extensive feedback from the preservice teachers and express their opinion on the assessment they took. The first-hand reactions from test-takers are illuminating; however, reactions like these are rarely shared with test-developers or inservice teachers. As both students’ and the inservice teacher’s interests were not at stake by critiquing the tests that were externally developed by those who are in-training, both parties were able to honestly and freely discuss the quality of assessment.

Final version received October 2022

The authors

Jiyoon Lee is a researcher and teacher educator in the TESOL program at the University of Maryland, Baltimore County in the United States. She conducts research on education stakeholders’ language assessment literacy and cross-discipline teacher education. She teaches graduate-level courses on language assessment and linguistics. She has also conducted workshops and professional development on assessment in the United States and abroad.

Email: [e-mail]

Natalia Goldenberg has been working as an ESOL teacher in Baltimore (the United States) for 20 years. She serves multilingual scholars and families from diverse linguistic and cultural backgrounds. She supports preservice and new teachers through internship supervision and mentoring. She believes in continuing professional growth and has achieved and maintained National Board Certification in English as a New Language. Her interests include reading, teaching writing to emergent multilingual students, and strengthening school culture.

Email: [e-mail]

Debby Pastrana Williams is an urban public school ESOL teacher located in Baltimore City, Maryland in the United States. She completed her TESOL K-12 master’s degree with high honors from the University of Maryland, Baltimore County in the United States. Ms Pastrana Williams has developed and facilitated district and school-wide professional developments to educate colleagues on best practices to support English language learners. Additionally, she has completed a postgraduate certificate in Dyslexia Studies and Language-Based Learning Disabilities.

Email: [e-mail]