Merrick / Maher Motivated Reinforcement Learning

Curious Characters for Multiuser Games
1. Auflage 2009
ISBN: 978-3-540-89187-1
Verlag: Springer Berlin Heidelberg
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 206 Seiten

Motivated learning is an emerging research field in artificial intelligence and cognitive modelling. Computational models of motivation extend reinforcement learning to adaptive, multitask learning in complex, dynamic environments - the goal being to understand how machines can develop new skills and achieve goals that were not predefined by human engineers. In particular, this book describes how motivated reinforcement learning agents can be used in computer games for the design of non-player characters that can adapt their behaviour in response to unexpected changes in their environment. This book covers the design, application and evaluation of computational models of motivation in reinforcement learning. The authors start with overviews of motivation and reinforcement learning, then describe models for motivated reinforcement learning. The performance of these models is demonstrated by applications in simulated game scenarios and a live, open-ended virtual world. Researchers in artificial intelligence, machine learning and artificial life will benefit from this book, as will practitioners working on complex, dynamic systems - in particular multiuser, online games.

Autoren/Hrsg.

Merrick, Kathryn E.

Maher, Mary Lou

Inhaltsverzeichnis

1;Preface;5
2;Acronyms;9
3;Contents;10
4;Part I Non-Player Characters and Reinforcement Learning;14
4.1;Chapter 1 Non-Player Characters in Multiuser Games;15
4.1.1;1.1 Types of Multiuser Games;16
4.1.1.1;1.1.1 Massively Multiplayer Online Role-Playing Games;16
4.1.1.2;1.1.2 Multiuser Simulation Games;17
4.1.1.3;1.1.3 Open-Ended Virtual Worlds;17
4.1.2;1.2 Character Roles in Multiuser Games;20
4.1.3;1.3 Existing Artificial Intelligence Techniques for Non- Player Characters in Multiuser Games;21
4.1.3.1;1.3.1 Reflexive Agents;21
4.1.3.2;1.3.2 Learning Agents;24
4.1.3.3;1.3.3 Evolutionary Agents;26
4.1.3.4;1.3.4 Smart Terrain;26
4.1.4;1.4 Summary;27
4.1.5;1.5 References;27
4.2;Chapter 2 Motivation in Natural and Artificial Agents;29
4.2.1;2.1 Defining Motivation;29
4.2.2;2.2 Biological Theories of Motivation;32
4.2.2.1;2.2.1 Drive Theory;32
4.2.2.2;2.2.2 Motivational State Theory;34
4.2.2.3;2.2.3 Arousal;35
4.2.3;2.3 Cognitive Theories of Motivation;38
4.2.3.1;2.3.1 Curiosity;38
4.2.3.2;2.3.2 Operant Theory;40
4.2.3.3;2.3.3 Incentive;41
4.2.3.4;2.3.4 Achievement Motivation;42
4.2.3.5;2.3.5 Attribution Theory;43
4.2.3.6;2.3.6 Intrinsic Motivation;45
4.2.4;2.4 Social Theories of Motivation;47
4.2.4.1;2.4.1 Conformity;47
4.2.4.2;2.4.2 Cultural Effect;48
4.2.4.3;2.4.3 Evolution;48
4.2.5;2.5 Combined Motivation Theories;49
4.2.5.1;2.5.1 Maslow’s Hierarchy of Needs;50
4.2.5.2;2.5.2 Existence Relatedness Growth Theory;50
4.2.6;2.6 Summary;51
4.2.7;2.7 References;52
4.3;Chapter 3 Towards Motivated Reinforcement Learning;56
4.3.1;3.1 Defining Reinforcement Learning;56
4.3.1.1;3.1.1 Dynamic Programming;58
4.3.1.2;3.1.2 Monte Carlo Methods;59
4.3.1.3;3.1.3 Temporal Difference Learning;60
4.3.2;3.2 Reinforcement Learning in Complex Environments;63
4.3.2.1;3.2.1 Partially Observable Environments;63
4.3.2.2;3.2.2 Function Approximation;64
4.3.2.3;3.2.3 Hierarchical Reinforcement Learning;65
4.3.3;3.3 Motivated Reinforcement Learning;68
4.3.3.1;3.3.1 Using a Motivation Signal in Addition to a Reward Signal;69
4.3.3.2;3.3.2 Using a Motivation Signal Instead of a Reward Signal;75
4.3.4;3.4 Summary;78
4.3.5;3.5 References;79
4.4;Chapter 4 Comparing the Behaviour of Learning Agents;82
4.4.1;4.1 Player Satisfaction;82
4.4.1.1;4.1.1 Psychological Flow;83
4.4.1.2;4.1.2 Structural Flow;84
4.4.2;4.2 Formalising Non-Player Character Behaviour;84
4.4.2.1;4.2.1 Models of Optimality for Reinforcement Learning;85
4.4.2.2;4.2.2 Characteristics of Motivated Reinforcement Learning;89
4.4.3;4.3 Comparing Motivated Reinforcement Learning Agents;92
4.4.3.1;4.3.1 Statistical Model for Identifying Learned Tasks;94
4.4.3.2;4.3.2 Behavioural Variety;94
4.4.3.3;4.3.3 Behavioural Complexity;96
4.4.4;4.4 Summary;97
4.4.5;4.5 References;98
5;Part II Developing Curious Characters Using Motivated Reinforcement Learning;100
5.1;Chapter 5 Curiosity, Motivation and Attention Focus;101
5.1.1;5.1 Agents in Complex, Dynamic Environments;101
5.1.1.1;5.1.1 States;103
5.1.1.2;5.1.2 Actions;104
5.1.1.3;5.1.3 Reward and Motivation;104
5.1.2;5.2 Motivation and Attention Focus;105
5.1.2.1;5.2.1 Observations;106
5.1.2.2;5.2.2 Events;108
5.1.2.3;5.2.3 Tasks and Task Selection;110
5.1.2.4;5.2.4 Experience-Based Reward as Cognitive Motivation;112
5.1.2.5;5.2.5 Arbitration Functions;118
5.1.2.6;5.2.6 A General Experience-Based Motivation Function;119
5.1.3;5.3 Curiosity as Motivation for Support Characters;121
5.1.3.1;5.3.1 Curiosity as Interesting Events;121
5.1.3.2;5.3.2 Curiosity as Interest and Competence;126
5.1.4;5.4 Summary;129
5.1.5;5.5 References;129
5.2;Chapter 6 Motivated Reinforcement Learning Agents;131
5.2.1;6.1 A General Motivated Reinforcement Learning Model;131
5.2.2;6.2 Algorithms for Motivated Reinforcement Learning;133
5.2.2.1;6.2.1 Motivated Flat Reinforcement Learning;133
5.2.2.2;6.2.2 Motivated Multioption Reinforcement Learning;136
5.2.2.3;6.2.3 Motivated Hierarchical Reinforcement Learning;141
5.2.3;6.3 Summary;143
5.2.4;6.4 References;144
6;Part III Curious Characters in Games;145
6.1;Chapter 7 Curious Characters for Multiuser Games;146
6.1.1;7.1 Motivated Reinforcement Learning for Support Characters in Massively Multiplayer Online Role-Playing Games;147
6.1.2;7.2 Character Behaviour in Small-Scale, Isolated Game Locations;150
6.1.2.1;7.2.1 Case Studies of Individual Characters;151
6.1.2.2;7.2.2 General Trends in Character Behaviour;154
6.1.3;7.3 Summary;157
6.1.4;7.4 References;158
6.2;Chapter 8 Curious Characters for Games in Complex, Dynamic Environments;159
6.2.1;8.1 Designing Characters That Can Multitask;160
6.2.1.1;8.1.1 Case Studies of Individual Characters;163
6.2.1.2;8.1.2 General Trends in Character Behaviour;164
6.2.2;8.2 Designing Characters for Complex Tasks;167
6.2.2.1;8.2.1 Case Studies of Individual Characters;167
6.2.2.2;8.2.2 General Trends in Character Behaviour;169
6.2.3;8.3 Games That Change While Characters Are Learning;171
6.2.3.1;8.3.1 Case Studies of Individual Characters;172
6.2.3.2;8.3.2 General Trends in Character Behaviour;175
6.2.4;8.4 Summary;177
6.2.5;8.5 References;178
6.3;Chapter 9 Curious Characters for Games in Second Life;179
6.3.1;9.1 Motivated Reinforcement Learning in Open-Ended Simulation Games;179
6.3.1.1;9.1.1 Game Design;180
6.3.1.2;9.1.2 Character Design;180
6.3.2;9.2 Evaluating Character Behaviour in Response to Game Play Sequences;184
6.3.2.1;9.2.1 Discussion;195
6.3.3;9.3 Summary;196
6.3.4;9.4 References;197
7;Part IV Future;198
7.1;Chapter 10 Towards the Future;199
7.1.1;10.1 Using Motivated Reinforcement Learning in Non-Player Characters;199
7.1.2;10.2 Other Gaming Applications for Motivated Reinforcement Learning;200
7.1.2.1;10.2.1 Dynamic Difficulty Adjustment;200
7.1.2.2;10.2.2 Procedural Content Generation;201
7.1.3;10.3 Beyond Curiosity;201
7.1.3.1;10.3.1 Biological Models of Motivation;201
7.1.3.2;10.3.2 Cognitive Models of Motivation;202
7.1.3.3;10.3.3 Social Models of Motivation;202
7.1.3.4;10.3.4 Combined Models of Motivation;202
7.1.4;10.4 New Models of Motivated Learning;203
7.1.4.1;10.4.1 Motivated Supervised Learning;203
7.1.4.2;10.4.2 Motivated Unsupervised Learning;204
7.1.5;10.5 Evaluating the Behaviour of Motivated Learning Agents;204
7.1.6;10.6 Concluding Remarks;204
7.1.7;10.7 References;205
8;Index;206