Hu / Yue Markov Decision Processes with Their Applications

1. Auflage 2007
ISBN: 978-0-387-36951-8
Verlag: Springer-Verlag
Format: PDF
Kopierschutz: Wasserzeichen (»Systemvoraussetzungen)

E-Book, Englisch, 297 Seiten

Put together by two top researchers in the Far East, this text examines Markov Decision Processes - also called stochastic dynamic programming - and their applications in the optimal control of discrete event systems, optimal replacement, and optimal allocations in sequential online auctions. This dynamic new book offers fresh applications of MDPs in areas such as the control of discrete event systems and the optimal allocations in sequential online auctions.

Autoren/Hrsg.

Hu, Qiying

Yue, Wuyi

Inhaltsverzeichnis

1;Contents;6
2;List of Figures;10
3;List of Tables;11
4;Preface;12
5;Acknowledgments;14
6;INTRODUCTION;15
6.1;1. A Brief Description of Markov Decision Processes;15
6.2;2. Overview of the Book;18
6.3;3. Organization of the Book;20
7;DISCRETE TIME MARKOV DECISION PROCESSES: TOTAL REWARD;24
7.1;1. Model and Preliminaries ;24
7.1.1;1.1 System Model;24
7.1.2;1.2 Some Concepts;25
7.1.3;1.3 Finiteness of the Reward;27
7.2;2. Optimality Equation;30
7.2.1;2.1 Validity of the Optimality Equation;30
7.2.2;2.2 Properties of the Optimality Equation;34
7.3;3. Properties of Optimal Policies;38
7.4;4. Successive Approximation;43
7.5;5. Sufficient Conditions;45
7.6;6. Notes and References;47
7.7;Problems;48
8;DISCRETE TIME MARKOV DECISION PROCESSES: AVERAGE CRITERION;52
8.1;1. Model and Preliminaries;52
8.2;2. Optimality Equation;56
8.2.1;2.1 Properties of ACOE and Optimal Policies;57
8.2.2;2.2 Sufficient Conditions;61
8.2.3;2.3 Recurrent Conditions;63
8.3;3. Optimality Inequalities;66
8.3.1;3.1 Conditions;67
8.3.2;3.2 Properties of ACOI and Optimal Policies;70
8.4;4. Notes and References;73
8.5;Problems;74
9;CONTINUOUS TIME MARKOV DECISION PROCESSES;75
9.1;1. A Stationary Model: Total Reward ;75
9.1.1;1.1 Model and Conditions;75
9.1.2;1.2 Model Decomposition;79
9.1.3;1.3 Some Properties;83
9.1.4;1.4 Optimality Equation and Optimal Policies;89
9.2;2. A Nonstationary Model: Total Reward;97
9.2.1;2.1 Model and Conditions;97
9.2.2;2.2 Optimality Equation;99
9.3;3. A Stationary Model: Average Criterion;107
9.4;4. Notes and References;113
9.5;Problems;114
10;SEMI-MARKOV DECISION PROCESSES;116
10.1;1. Model and Conditions ;116
10.1.1;1.1 Model;116
10.1.2;1.2 Regular Conditions;118
10.1.3;1.3 Criteria;121
10.2;2. Transformation;122
10.2.1;2.1 Total Reward;123
10.2.2;2.2 Average Criterion;126
10.3;3. Notes and References;130
10.4;Problems;131
11;MARKOV DECISION PROCESSES IN SEMI- MARKOV ENVIRONMENTS;132
11.1;1. Continuous Time Markov Decision Processes in Semi- Markov Environments;132
11.1.1;1.1 Model;132
11.1.2;1.2 Optimality Equation;138
11.1.3;1.3 Approximation byWeak Convergence;148
11.1.4;1.4 Markov Environment;151
11.1.5;1.5 Phase Type Environment;154
11.2;2. Semi-Markov Decision Processes in Semi-Markov Environments;159
11.2.1;2.1 Model;159
11.2.2;2.2 Optimality Equation;163
11.2.3;2.3 Markov Environment;169
11.3;3. Mixed Markov Decision Processes in Semi-Markov Environments;171
11.3.1;3.1 Model;171
11.3.2;3.2 Optimality Equation;174
11.3.3;3.3 Markov Environment;181
11.4;4. Notes and References;185
11.5;Problems;186
12;OPTIMAL CONTROL OF DISCRETE EVENT SYSTEMS: I;187
12.1;1. System Model;187
12.2;2. Optimality;190
12.2.1;2.1 Maximum Discounted Total Reward;192
12.2.2;2.2 Minimum Discounted Total Reward;196
12.3;3. Optimality in Event Feedback Control;196
12.4;4. Link to Logic Level;199
12.5;5. Resource Allocation System;204
12.6;6. Notes and References;211
12.7;Problems;212
13;OPTIMAL CONTROL OF DISCRETE EVENT SYSTEMS: II;213
13.1;1. System Model;213
13.2;2. Optimality Equation and Optimal Supervisors;217
13.3;3. Language Properties;223
13.4;4. System Based on Automaton;225
13.5;5. Supervisory Control Problems;228
13.5.1;5.1 Event Feedback Control;228
13.5.2;5.2 State Feedback Control;232
13.6;6. Job-Matching Problem;233
13.7;7. Notes and References;240
13.8;Problems;240
14;OPTIMAL REPLACEMENT UNDER STOCHASTIC ENVIRONMENTS;242
14.1;1. Optimal Replacement: Discrete Time ;243
14.1.1;1.1 Problem and Model;243
14.1.2;1.2 Total Cost Criterion;247
14.1.3;1.3 Average Criterion;250
14.2;2. Optimal Replacement: Semi-Markov Processes ;253
14.2.1;2.1 Problem;253
14.2.2;2.2 Optimal Control Limit Policies;256
14.2.3;2.3 Markov Environment;259
14.2.4;2.4 Numerical Example;267
14.3;3. Notes and References;269
14.4;Problems;271
15;OPTIMAL ALLOCATION IN SEQUENTIAL ONLINE AUCTIONS;273
15.1;1. Problem and Model;273
15.2;2. Analysis for Private Reserve Price;275
15.3;3. Analysis for Announced Reserve Price;279
15.4;4. Monotone Properties;281
15.5;5. Numerical Results;290
15.6;6. Notes and References;292
15.7;Problems;293
16;References;295
17;Index;303