Ifenthaler / Pirnay-Dummer / Seel Computer-Based Diagnostics and Systematic Analysis of Knowledge

1. Auflage 2010
ISBN: 978-1-4419-5662-0
Verlag: Springer
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 371 Seiten

What is knowledge? How can it be successfully assessed? How can we best use the results? As questions such as these continue to be discussed and the learning sciences continue to deal with expanding amounts of data, the challenge of applying theory to diagnostic methods takes on more complexity. Computer-Based Diagnostics and Systematic Analysis of Knowledge meets this challenge head-on as an international panel of experts reviews current and emerging assessment methodologies in the psychological and educational arenas. Emphasizing utility, effectiveness, and ease of interpretation, contributors critically discuss practical innovations and intriguing possibilities (including mental representations, automated knowledge visualization, modeling, and computer-based feedback) across fields ranging from mathematics education to medicine. These contents themselves model the steps of systematic inquiry, from theoretical construct to real-world application: Historical and theoretical foundations for the investigation of knowledge Current opportunities for understanding knowledge empirically Strategies for the aggregation and classification of knowledge Tools and methods for comparison and empirical testing Data interfaces between knowledge assessment tools Guidance in applying research results to particular fields Researchers and professionals in education psychology, instructional technology, computer science, and linguistics will find Computer-Based Diagnostics and Systematic Analysis of Knowledge a stimulating guide to a complex present and a rapidly evolving future.

Autoren/Hrsg.

Pirnay-Dummer, Pablo

Ifenthaler, Dirk

Seel, Norbert M.

Inhaltsverzeichnis

1;Preface;6
1.1;Elicitation of Knowledge;7
1.2;Aggregation and Classification of Knowledge;7
1.3;Comparison and Empirical Testing of Strategies;7
1.4;Application of Results;7
1.5;Acknowledgements;7
2;Contents;10
3;Contributors;14
4;About the Authors;16
5;Reviewers;22
6;Part I Elicitation of Knowledge;23
6.1;Intermezzo 1 -- To Be Moved by Knowledge: Moving Knowledge MovesKnowledgeAboutKnowing;23
6.2;1 Essentials of Computer-Based Diagnostics of Learning and Cognition;25
6.2.1;1.1 Introduction;25
6.2.2;1.2 Diagnostics and Diagnosis in The Area Of Education and Instruction;27
6.2.3;1.3 Responsive Methods of Measurement and Assessment;28
6.2.4;1.4 Constructive Methods of Knowledge Diagnosis;29
6.2.5;1.5 The Role Of External Representations;30
6.2.6;1.6 Computer-Based and Agent-Based Knowledge Diagnosis;33
6.2.7;1.7 Preview;35
6.2.8;References;35
6.3;2 A Functional View Toward Mental Representations;37
6.3.1;2.1 Representation in General;37
6.3.1.1;2.1.1 Mental Representations;38
6.3.1.2;2.1.2 The Vehicle;39
6.3.1.3;2.1.3 The Representandum;39
6.3.1.4;2.1.4 The Subject;40
6.3.1.5;2.1.5 The Triple-Digit Relation;40
6.3.1.6;2.1.6 Types of Mental Representation;41
6.3.2;2.2 Related Notions;43
6.3.2.1;2.2.1 ''Explained by '''';43
6.3.2.2;2.2.2 ''Structured by '''';43
6.3.2.3;2.2.3 Misrepresentation;44
6.3.3;2.3 How to Operate on Representations;44
6.3.3.1;2.3.1 How Do You Know that Someone Is Able to Generate and Use Conceptual Representations?;45
6.3.4;References;46
6.4;3 Mental Representations and Their Analysis: An Epistemological Perspective;48
6.4.1;3.1 Introduction;48
6.4.1.1;3.1.1 The Nature of Learning;48
6.4.1.2;3.1.2 Learning Complex Problem-Solving Tasks and Skills;49
6.4.2;3.2 Mental Models;54
6.4.3;3.3 Mental Model Assessments and Learning Progress;55
6.4.4;3.4 An Epistemological Perspective;58
6.4.5;3.5 Concluding Remarks;59
6.4.6;References;60
6.5;4 Multi-decision Approaches for Eliciting Knowledge Structure;62
6.5.1;4.1 Knowledge and Knowledge Structure;62
6.5.2;4.2 Relatedness Data and Its Analysis and Representation;63
6.5.3;4.3 Alternative Approaches to Elicit Relatedness Data;66
6.5.3.1;4.3.1 The Role of Context;67
6.5.3.2;4.3.2 Computer-Based Listwise and Sorting Multi-decision Approaches;69
6.5.3.3;4.3.3 The Effects of Headings on Knowledge Structure;69
6.5.3.4;4.3.4 Listwise and Sorting Approaches Compared to the Pairwise Approach;71
6.5.3.5;4.3.5 Sorting and Listwise Combined Approach;75
6.5.4;4.4 Summary and Conclusion;78
6.5.5;References;79
6.6;5 The Problem of Knowledge Elicitation from the Experts Point of View;81
6.6.1;5.1 Introduction;81
6.6.2;5.2 Description of the System and Knowledge Elicitation;82
6.6.3;5.3 Language Skills Database;87
6.6.4;5.4 Adaptive Fuzzy E-Learning Subsystems;88
6.6.5;5.5 Supervised Learning Schema;90
6.6.6;5.6 Conclusions;92
6.6.7;References;92
7;Part II Aggregation and Classification of Knowledge;94
7.1;Intermezzo 2 -- Artefacts of Thought: Properties and KindsofRe-representations;94
7.2;6 Automated Knowledge Visualization and Assessment;96
7.2.1;6.1 Introduction;96
7.2.2;6.2 Applying Current Computer Technology;97
7.2.3;6.3 Automated Tools;99
7.2.3.1;6.3.1 Mitocar;99
7.2.3.1.1;6.3.1.1 Two Phases of Data Collection;99
7.2.3.1.2;6.3.1.2 Graphical Re-representation of the Model;102
7.2.3.1.3;6.3.1.3 Additional Descriptive Elicitation Modules;103
7.2.3.1.4;6.3.1.4 Automated Report Engine;105
7.2.3.2;6.3.2 T-MITOCAR;107
7.2.3.2.1;6.3.2.1 Preparing the Text;107
7.2.3.2.2;6.3.2.2 Tokenizing;107
7.2.3.2.3;6.3.2.3 Tagging;107
7.2.3.2.4;6.3.2.4 Stemming;108
7.2.3.2.5;6.3.2.5 Fetching the Most Frequent Concepts from the Text;108
7.2.3.2.6;6.3.2.6 Sum of Distances: Determining Pairwise Associatedness;108
7.2.3.2.7;6.3.2.7 Determining the Weights;109
7.2.3.2.8;6.3.2.8 De-stemming;109
7.2.3.2.9;6.3.2.9 Writing the Model to the List Form;109
7.2.3.2.10;6.3.2.10 Example from Wikipedia Texts (Economy);109
7.2.3.2.11;6.3.2.11 How to Use T-MITOCAR;111
7.2.3.2.12;6.3.2.12 Applications;112
7.2.3.3;6.3.3 T-MITOCAR Artemis;113
7.2.3.3.1;6.3.3.1 Input Formats and Interface;113
7.2.3.3.2;6.3.3.2 Output Format of the Knowledge Map;114
7.2.3.4;6.3.4 SMD Technology;114
7.2.3.4.1;6.3.4.1 Phase 1: Input;114
7.2.3.4.2;6.3.4.2 Phase 2: Analysis Specification;117
7.2.3.4.3;6.3.4.3 Phase 3: Quantitative Analysis Output;117
7.2.3.4.4;6.3.4.4 Phase 4: Standardized Graphical Output;117
7.2.3.5;6.3.5 Model Comparison;118
7.2.3.6;6.3.6 Comparison Measures;120
7.2.3.6.1;6.3.6.1 Surface Matching;120
7.2.3.6.2;6.3.6.2 Graphical Matching;120
7.2.3.6.3;6.3.6.3 Structural Matching;121
7.2.3.6.4;6.3.6.4 Gamma Matching;121
7.2.3.6.5;6.3.6.5 Concept Matching;121
7.2.3.6.6;6.3.6.6 Propositional Matching;121
7.2.3.6.7;6.3.6.7 Balanced Semantic Matching;122
7.2.3.6.8;6.3.6.8 Triangulation of Types of Expertise;122
7.2.3.7;6.3.7 HIMATT;122
7.2.4;6.4 AKOVIA;124
7.2.4.1;6.4.1 Foundation and Design;125
7.2.4.2;6.4.2 AKOVIA Input;125
7.2.4.2.1;6.4.2.1 Input from Graphs (List Form);126
7.2.4.2.2;6.4.2.2 Input from Text;127
7.2.4.2.3;6.4.2.3 Mixed Format Input;127
7.2.4.3;6.4.3 Common Model Data Frame;128
7.2.4.4;6.4.4 Analysis and Scripting;128
7.2.4.4.1;6.4.4.1 Visualize;129
7.2.4.4.2;6.4.4.2 Ganalyze;129
7.2.4.4.3;6.4.4.3 Compare;129
7.2.4.4.4;6.4.4.4 Aggregate;130
7.2.4.5;6.4.5 Upload, Feedback, and Analysis;130
7.2.4.6;6.4.6 Server Topology;130
7.2.4.7;6.4.7 Data Warehousing;131
7.2.5;6.5 Applications And Future Perspectives;131
7.2.5.1;6.5.1 Applications;132
7.2.5.2;6.5.2 Future Perspectives;132
7.2.6;References;133
7.3;7 Deriving Individual andINTtie;Group Knowledge Structure fromINTtie;Network Diagrams andINTtie;from Essays;135
7.3.1;7.1 Introduction;135
7.3.2;7.2 Pathfinder Network Analysis;135
7.3.3;7.3 Network Diagrams and Knowledge Structure;136
7.3.3.1;7.3.1 ALA-Mapper Investigations;137
7.3.3.2;7.3.2 Rubrics and Network Diagram Scores;139
7.3.4;7.4 Essays and Knowledge Structure;140
7.3.4.1;7.4.1 Sentence Aggregate Approach;141
7.3.4.2;7.4.2 Linear Aggregate Approach;142
7.3.5;7.5 Next Steps;145
7.3.6;References;147
7.4;8 A Self-Organising Systems Approach to History-Enriched Digital Objects;149
7.4.1;8.1 Self-Organising Systems;149
7.4.2;8.2 Social Software;151
7.4.2.1;8.2.1 Social Software for Education;152
7.4.3;8.3 Text Signalling;153
7.4.4;8.4 History-Enriched Digital Objects;154
7.4.5;8.5 Summary;155
7.4.6;8.6 The Design of CoREAD;155
7.4.7;8.7 The Study;157
7.4.8;8.8 Method;157
7.4.9;8.9 Results;158
7.4.9.1;8.9.1 Descriptive Statistics;159
7.4.9.2;8.9.2 The Author's Summary;160
7.4.9.2.1;8.9.2.1 The Author's Summary Reproduced;160
7.4.9.3;8.9.3 Students' Highlights;161
7.4.9.3.1;8.9.3.1 Differences Between the Text and Author Comparisons;161
7.4.9.3.2;8.9.3.2 Trend over Time;161
7.4.9.3.3;8.9.3.3 Correlational Analyses;161
7.4.9.4;8.9.4 Students' Written Summaries;163
7.4.9.4.1;8.9.4.1 Differences Between the Text and Author Comparisons;163
7.4.9.4.2;8.9.4.2 Trend over Time;164
7.4.9.4.3;8.9.4.3 Correlational Analyses;164
7.4.9.4.4;8.9.4.4 Multiple Regression Analyses;164
7.4.9.5;8.9.5 Case Studies;165
7.4.9.5.1;8.9.5.1 Best Summary When Compared to the Flynn Effect Text # Participant #14;166
7.4.9.5.2;8.9.5.2 Worst Summary When Compared to the Flynn Effect Text # Participant #1;167
7.4.9.5.3;8.9.5.3 Best Summary When Compared to the Author#s Summary # Particpant #4;168
7.4.9.5.4;8.9.5.4 Worst Summary When Compared to the Author#s Summary # Participant #40;169
7.4.10;8.10 Discussion;170
7.4.10.1;8.10.1 Social Software for Assessment and Feedback;170
7.4.10.1.1;8.10.1.1 Assessment;170
7.4.10.1.2;8.10.1.2 Feedback;171
7.4.10.2;8.10.2 Limitations and Future Work;172
7.4.10.2.1;8.10.2.1 Limitations of Highlighting;172
7.4.10.2.2;8.10.2.2 No Trend over Time;173
7.4.10.2.3;8.10.2.3 Effects of Text Length on LSA Scores;173
7.4.11;8.11 Conclusion;173
7.4.12;References;174
7.5;9 Performance Categories: Task-Diagnostic Techniquesand Interfaces;177
7.5.1;9.1 Introduction;177
7.5.2;9.2 Tasks;177
7.5.2.1;9.2.1 Tasks, Outcomes, and Processing;178
7.5.2.2;9.2.2 Tasks at Work;178
7.5.2.3;9.2.3 A Continuum of Tasks;179
7.5.3;9.3 Diagnostic Environments;180
7.5.3.1;9.3.1 Task Success;181
7.5.4;9.4 The Diagnostic Continuum;181
7.5.4.1;9.4.1 Prescribed Task Diagnosis;182
7.5.4.2;9.4.2 Discretionary Task Diagnosis;183
7.5.5;9.5 Delivery Mechanisms for Diagnosis in Prescribed Tasks;184
7.5.5.1;9.5.1 Computer-Based Training Diagnosis;184
7.5.5.2;9.5.2 Work Task Diagnosis;185
7.5.5.3;9.5.3 Agent-Based Diagnosis;186
7.5.6;9.6 Delivery Mechanisms for Diagnosis in Discretionary Tasks;187
7.5.6.1;9.6.1 Discretionary Simulations;187
7.5.6.2;9.6.2 Consultant Agents;188
7.5.7;9.7 Conclusion;189
7.5.8;References;190
8;Part III Comparison and Empirical Testing Strategies;192
8.1;Intermezzo 3 -- The Inner Workings of Knowledge and Its Structure: Reasoning, Comparison, Testing, Evaluation, Decision,andAc;192
8.2;10 Graphs and Networks;194
8.2.1;10.1 Graphs as Representations of Binary Relations;194
8.2.2;10.2 Graphs and Matrices;197
8.2.3;10.3 Connectivity;199
8.2.4;10.4 Graph Isomorphism;200
8.2.5;10.5 Networks;201
8.2.6;10.6 Drawing Graphs;203
8.2.7;References;204
8.3;11 Abductive Reasoning and Similarity: Some ComputationalTools;206
8.3.1;11.1 Introduction;206
8.3.1.1;11.1.1 Abductive Reasoning;207
8.3.1.2;11.1.2 The Importance of Novelty;210
8.3.1.3;11.1.3 Approaches to Understanding the Generation of Hypotheses;211
8.3.1.4;11.1.4 Optimizing Versus Satisficing;212
8.3.2;11.2 Generating and Evaluating Hypotheses;213
8.3.2.1;11.2.1 Constraints on Abduction;213
8.3.2.2;11.2.2 Similarity in Abductive Inference;216
8.3.3;11.3 Random Vectors and Pathfinder Networks as Aids for Abduction from Text;217
8.3.4;11.4 Predicting Discoveries;223
8.3.5;11.5 Conclusions;225
8.3.6;References;225
8.4;12 Scope of Graphical Indices in Educational Diagnostics;229
8.4.1;12.1 Introduction;229
8.4.2;12.2 Graphs as External Knowledge Representation;231
8.4.2.1;12.2.1 Basics of Graph Theory;232
8.4.2.2;12.2.2 Measures of Graph Theory;232
8.4.2.3;12.2.3 Measures Beyond Graph Theory;235
8.4.2.4;12.2.4 Implementation of Graphical Indices for Educational Diagnostics;236
8.4.3;12.3 Empirical Studies;237
8.4.3.1;12.3.1 Development of Cognitive Structures;238
8.4.3.2;12.3.2 Feedback for Improving Expert Performance;241
8.4.3.3;12.3.3 Between-Domain Distinguishing Features of Cognitive Structures;244
8.4.4;12.4 Conclusion;245
8.4.5;References;247
8.5;13 Complete Structure Comparison;251
8.5.1;13.1 Knowledge and Structure;251
8.5.2;13.2 Retracing Knowledge Structure;253
8.5.3;13.3 Completeness and Explanatory Power;254
8.5.4;13.4 Simple Structures and a Preliminary Structural Notation;255
8.5.4.1;13.4.1 Complete Structural Traces;257
8.5.4.2;13.4.2 Downtrace;259
8.5.4.3;13.4.3 Structural Matching Similarity Measure;261
8.5.5;13.5 Studies 1 and 2: Trace-Based Structural Complexity Measure;261
8.5.5.1;13.5.1 Methods;262
8.5.5.2;13.5.2 Study 1: In the Field of Learning and Instruction;263
8.5.5.3;13.5.3 Study 2: In the Field of Economics;264
8.5.5.4;13.5.4 Discussion of Studies 1 and 2;265
8.5.6;13.6 Study 3: Technological Study on the Sensitivity of Structural Matching;265
8.5.6.1;13.6.1 Methods;265
8.5.6.2;13.6.2 Results;266
8.5.6.3;13.6.3 Post Hoc Analysis;267
8.5.6.4;13.6.4 Discussion of Study 3;267
8.5.7;13.7 Study 4: Empirical Study on the Semantic Interference with Structural Matching;268
8.5.7.1;13.7.1 Methods;268
8.5.7.2;13.7.2 Results;269
8.5.7.3;13.7.3 Post Hoc Analysis;270
8.5.7.4;13.7.4 Discussion of Study 4;271
8.5.8;13.8 Comparison to Heuristic Measures of Structure;271
8.5.9;13.9 Conclusion;272
8.5.10;References;273
9;Part IV Application of Obtained Results;275
9.1;Intermezzo 4 -- Using Knowledge to Support Knowing;275
9.2;14 Computer-Based Feedback for Computer-Based Collaborative Problem Solving;277
9.2.1;14.1 Introduction;277
9.2.1.1;14.1.1 Effects of Visual and Verbal Feedback;281
9.2.1.2;14.1.2 Methodology;282
9.2.1.3;14.1.3 Networked Knowledge Mapping System;282
9.2.1.4;14.1.4 Simulated World Wide Web Environment;283
9.2.1.5;14.1.5 Feedback;284
9.2.1.6;14.1.6 Participants;284
9.2.2;14.2 Measures;285
9.2.2.1;14.2.1 Group Outcome Measures;285
9.2.2.2;14.2.2 Information Seeking and Feedback Behavior Measures;285
9.2.2.3;14.2.3 Teamwork Process Measures;286
9.2.3;14.3 Procedure;286
9.2.3.1;14.3.1 Teamwork Questionnaire;287
9.2.3.2;14.3.2 Task Instructions and Search Strategies Training;287
9.2.3.3;14.3.3 Collaborative Group Task 1;287
9.2.3.4;14.3.4 After-Action Review Feedback;287
9.2.3.5;14.3.5 Collaborative Group Task 2;289
9.2.3.6;14.3.6 Debriefing;289
9.2.4;14.4 Data Analysis;289
9.2.5;14.5 Results and Discussion;289
9.2.5.1;14.5.1 The Effect of AAR on Team Map Scores;289
9.2.5.2;14.5.2 The Effect of AAR on Search Scores;290
9.2.5.3;14.5.3 The Effect of AAR on Communication Scores;291
9.2.6;14.6 Summary and Conclusions;292
9.2.7;14.7 Summary of Chapter;292
9.2.8;References;293
9.3;15 Modeling, Assessing, and Supporting Key Competencies Within Game Environments;296
9.3.1;15.1 Introduction;296
9.3.1.1;15.1.1 Purpose;297
9.3.1.2;15.1.2 Where We Are;298
9.3.1.2.1;15.1.2.1 Disengaged Students;298
9.3.1.2.2;15.1.2.2 The Shrinking World;298
9.3.1.3;15.1.3 Where We Should Be Heading;299
9.3.2;15.2 Assessment Methodology: Evidence-Centered Design;300
9.3.2.1;15.2.1 ECD Models;300
9.3.2.1.1;15.2.1.1 Competency Model;301
9.3.2.1.2;15.2.1.2 Evidence Model;301
9.3.2.1.3;15.2.1.3 Task Model;301
9.3.2.1.4;15.2.1.4 Design and Diagnosis;302
9.3.2.2;15.2.2 Stealth Assessment;302
9.3.2.3;15.2.3 Systems Thinking;303
9.3.2.3.1;15.2.3.1 Definitions of Systems Thinking;303
9.3.2.3.2;15.2.3.2 Systems Thinking and Its Role in Education;304
9.3.2.3.3;15.2.3.3 The Competency Model of Systems Thinking;304
9.3.3;15.3 Application of the Stealth Assessment Approach;308
9.3.3.1;15.3.1 Quest Atlantis: Taiga Park;308
9.3.3.2;15.3.2 ECD Models Applied to Taiga;310
9.3.3.2.1;15.3.2.1 Tools to Automatically Assess Causal Diagrams;313
9.3.3.2.2;15.3.2.2 Adding Stealth Assessment to Taiga;315
9.3.4;15.4 Summary and Discussion;319
9.3.5;References;322
9.4;16 A Methodology for Assessing Elicitation of Knowledge in Complex Domains: Identifying Conceptual Representations of Ill-Structured Problems in Medical Diagnosis;325
9.4.1;16.1 Introduction;325
9.4.1.1;16.1.1 Assessing Learning in Complex Domains;325
9.4.1.2;16.1.2 Assessing the Ability to Solve Ill-Defined Problems;327
9.4.1.3;16.1.3 Assessing Progress in Complex Problem Solving in DEEP;328
9.4.2;16.2 Methods;329
9.4.2.1;16.2.1 Research Design and Questions;329
9.4.2.2;16.2.2 Research Methodology;330
9.4.2.3;16.2.3 Problem Scenarios;330
9.4.2.4;16.2.4 Participants;330
9.4.2.5;16.2.5 Data Collection Process;332
9.4.2.6;16.2.6 Data Analysis;333
9.4.3;16.3 Results and Analysis;336
9.4.3.1;16.3.1 Comparisons of Expert and Novice Responses;339
9.4.4;16.4 Conclusion;342
9.4.4.1;16.4.1 Medical Domain Issues;343
9.4.4.2;16.4.2 Further Work on the DEEP Tools and Analysis Methodology;344
9.4.5;References;345
9.5;17 Selection ofINTtie;Team Interventions Based onINTtie;Mental Model Sharedness Levels Measured byINTtie;theINTtie;Team Assessment andINTtie;Diagnostic Instrument (TADI);349
9.5.1;17.1 Introduction;349
9.5.2;17.2 Team Cognition;350
9.5.3;17.3 Team Assessment and Diagnostic Instrument;352
9.5.4;17.4 Data Collection and Analysis;353
9.5.5;17.5 Interventions;355
9.5.5.1;17.5.1 Intervention Decision Making;356
9.5.5.1.1;17.5.1.0 Phase 1: Determining the Need for Consensus-Building (CB) Interventions;357
9.5.5.1.2;17.5.1.0 Phase 2: Determining the Need for Team Improvement Planning (TIP) Interventions;359
9.5.5.2;17.5.2 Intervention Types;360
9.5.5.2.1;17.5.2.0 Consensus Building (CB) Interventions;360
9.5.5.2.2;17.5.2.0 Team Improvement Planning (TIP) Interventions;361
9.5.5.3;17.5.3 Intervention Focusing on Consensus Building and Team Improvement Planning;362
9.5.6;17.6 Extension of TADI;363
9.5.7;17.7 Application of the TADI Measures for Selection of Team Interventions;363
9.5.7.1;17.7.1 Consider the TADI Similarity Measure;364
9.5.7.2;17.7.2 Look at the Range of the Similarity and Degree Measures;364
9.5.7.3;17.7.3 Examine All of the TADI Factors;364
9.5.7.4;17.7.4 Focus on One Team at a Time;365
9.5.8;References;366
10;Author Index;369
11;Subject Index;379