Harsch Calibrating Standards-based Assessment Tasks for English as a First Foreign Language. Standard-setting Procedures in Germany

1. Auflage 2010
ISBN: 978-3-8309-7299-0
Verlag: Waxmann Verlag GmbH
Format: PDF
Kopierschutz: Adobe DRM (»Systemvoraussetzungen)

E-Book, Englisch, 167 Seiten

This report is the second in a multi-part technical report series describing the development, calibration and validation of standards-based tests for English as a first foreign language at the Institute for Educational Progress (Institut zur Qualitätsentwicklung im Bildungswesen, IQB) in Berlin, Germany. It details the empirical basis of the calibration of the test item pool and criterion-referenced standard-setting procedures. The aim is to make transparent the decisions, methods and procedures which led to the setting of cut-scores in alignment with the National Educational Standards (NES) (i.e., the Länderübergreifende Bildungsstandards) and the Common European Framework of Reference for Languages (CEF) for English as a first foreign language. Standards were set for the lower secondary school level of Hauptschulabschluss and Mittlerer Bildungsabschluss.
This second report describes the process of relating the standards-based proficiency tests to the CEF levels, delineating the purpose of the tests, the aims of the standard-setting procedures, the rationale of the chosen procedures, and the use of the Manual for Linking Language Examinations to the CEF. This is followed by a detailed description of the two standard-setting methods employed, the Bookmark method and the computer-assisted Criterion Mapping method—developed and implemented by the Berkeley Evaluation and Assessment Research (BEAR) Center, University of California. This volume provides information on how the standard-setting sessions were conducted in collaboration with BEAR Center, and describes procedures, data gathering, and issues and problems which arose during the course of the study. The report concludes with a synthesis of the standard-setting study’s results, and discusses the implications of how these results are reported and presented to stakeholders and policy makers.

Autoren/Hrsg.

Harsch, Claudia

Inhaltsverzeichnis

1;Table of Contents;6
2;Preamble;8
3;Chapter 1: Setting Standards in Line with the Common European Framework of Reference;10
3.1;1.1 Educational Standards and Standards-based Assessment;12
3.2;1.2 Relating IQB Tests to CEF-levels;19
3.3;1.3 Rationale for Selecting the Item Pool;27
3.4;1.4 Description of Panelists;29
3.5;1.5 Standard-setting Procedure at IQB;31
3.6;1.6 Research into Validity of the Standard-setting Procedure ( DFG Research Grant);35
4;Chapter 2: Standard- setting Item Pool;36
4.1;2.1 Overview of Pilot Study Design;38
4.2;2.2 Methodology;41
4.3;2.3 Analyses for Reading and Listening Comprehension Scales;42
4.4;2.4 Analyses for Written Expression Scale;46
4.5;2.5 Descriptive Summary Statistics;48
4.6;2.6 Results from Multi-faceted Rasch Analyses ;51
5;Chapter 3: The Bookmark Standard- setting Method;56
5.1;3.1 Background;58
5.2;3.2 Bookmark Standard-setting Method;59
5.3;3.3 Standard-setting Workshop Results;65
6;Chapter 4: The Standard- setting Criterion Mapping Method;68
6.1;4.1 Background;70
6.2;4.2 Preparation;73
7;Chapter 5: Procedural Standard- setting Issues;80
7.1;5.1 Criterion Mapping;82
7.2;5.2 Bookmark;92
7.3;5.3 Summary and Conclusions;105
8;Chapter 6: Comparison and Synthesis of Multiple Standard- setting Methods and Panels;106
8.1;6.1 Setting Educational Standards;108
8.2;6.2 The Standard-setting Study;108
8.3;6.3 General Conclusions;111
8.4;6.4 Synthesizing and Reporting Results;111
9;Appendices;120
9.1;Appendix A;122
9.2;Appendix B;125
9.3;Appendix C;130
9.4;Appendix D;147
9.5;Appendix E;149
9.6;Appendix F;152
9.7;Appendix G;155
9.8;Appendix H;157
9.9;Appendix I;159