Tian / Zhao Optimized Cloud Resource Management and Scheduling

Theories and Practices
1. Auflage 2014
ISBN: 978-0-12-801645-9
Verlag: Elsevier Science & Techn.
Format: EPUB
Kopierschutz: 6 - ePub Watermark

E-Book, Englisch, 284 Seiten

Optimized Cloud Resource Management and Scheduling identifies research directions and technologies that will facilitate efficient management and scheduling of computing resources in cloud data centers supporting scientific, industrial, business, and consumer applications. It serves as a valuable reference for systems architects, practitioners, developers, researchers and graduate level students. - Explains how to optimally model and schedule computing resources in cloud computing - Provides in depth quality analysis of different load-balance and energy-efficient scheduling algorithms for cloud data centers and Hadoop clusters - Introduces real-world applications, including business, scientific and related case studies - Discusses different cloud platforms with real test-bed and simulation tools

Dr. Wenhong Tian has a PhD from Computer Science Department of North Carolina State University(NCSU) and did post-doc with joint funding from Ork Ridge National Lab and NCSU. He is now an associate professor at University of Electronic Science and Technology of China. His research interests include modeling and performance analysis of communication networks, Cloud computing and bio-computing. He has published more than 40 journal /conference papers in related areas.

Autoren/Hrsg.

Tian, Wenhong

Zhao, Yong

Inhaltsverzeichnis

1;Front Cover;1
2;Optimized Cloud Resource Management and Scheduling;4
3;Copyright Page;5
4;Contents;6
5;Foreword;10
6;Preface;12
7;About the Authors;16
8;Acknowledgments;18
9;1 An Introduction to Cloud Computing;20
9.1;Main Contents of this Chapter;20
9.2;1.1 The background of Cloud computing;20
9.3;1.2 Cloud computing is an integration of other advanced technologies;22
9.3.1;1.2.1 Parallel computing;23
9.3.2;1.2.2 Grid computing;24
9.3.3;1.2.3 Utility computing;25
9.3.4;1.2.4 Ubiquitous computing;25
9.3.5;1.2.5 Software as a service;26
9.3.6;1.2.6 Virtualization technology;26
9.4;1.3 The driving forces of Cloud computing;27
9.5;1.4 The development status and trends of Cloud computing;27
9.6;1.5 The classification of Cloud computing applications;29
9.6.1;1.5.1 Classification by service type;29
9.6.2;1.5.2 Classification by deployment method;30
9.7;1.6 The different roles in the Cloud computing industry chain;31
9.8;1.7 The main features and technical challenges of Cloud computing;32
9.8.1;1.7.1 The main features of Cloud computing;32
9.8.2;1.7.2 Challenging issues;32
9.9;Summary;34
9.10;References;34
10;2 Big Data Technologies and Cloud Computing;36
10.1;Main Contents of this Chapter;36
10.2;2.1 The background and definition of big data;36
10.3;2.2 Big data problems;39
10.3.1;2.2.1 The problem of speed;39
10.3.2;2.2.2 The type and architecture problem;40
10.3.3;2.2.3 Volume and flexibility problems;40
10.3.4;2.2.4 The cost problem;41
10.3.5;2.2.5 The value mining problem;42
10.3.6;2.2.6 The security and privacy problem;43
10.3.7;2.2.7 Interoperability and data sharing issues;44
10.4;2.3 The dialectical relationship between Cloud computing and big data;45
10.5;2.4 Big data technologies;47
10.5.1;2.4.1 Infrastructure support;48
10.5.2;2.4.2 Data acquisition;50
10.5.3;2.4.3 Data storage;51
10.5.4;2.4.4 Data computing;55
10.5.4.1;2.4.4.1 Offline batch computing;55
10.5.4.2;2.4.4.2 Real-time interactive computing;57
10.5.4.3;2.4.4.3 Streaming computing;59
10.5.5;2.4.5 Data presentation and interaction;61
10.5.6;2.4.6 Related work;64
10.6;Summary;65
10.7;Acknowledgments;65
10.8;References;66
11;3 Resource Modeling and Definitions for Cloud Data Centers;70
11.1;Main Contents of this Chapter;70
11.2;3.1 Resource models in Cloud data centers;70
11.3;3.2 Data center resources;70
11.4;3.3 Categories of Cloud data center resources;73
11.4.1;3.3.1 Properties and operations of various resources;73
11.4.1.1;3.3.1.1 Physical servers (PMs);73
11.4.1.1.1;3.3.1.1.1 The main properties of a physical server;73
11.4.1.1.2;3.3.1.1.2 Physical server states;74
11.4.1.1.3;3.3.1.1.3 Main operations of a physical server;75
11.4.1.1.4;3.3.1.1.4 Server operation error;75
11.4.1.2;3.3.1.2 Physical server cluster;75
11.4.1.2.1;3.3.1.2.1 Main properties of a physical server cluster;75
11.4.1.2.2;3.3.1.2.2 States of a physical server cluster;76
11.4.1.2.3;3.3.1.2.3 Operations of a physical server cluster;76
11.4.1.2.4;3.3.1.2.4 Physical server errors;76
11.4.1.3;3.3.1.3 Virtual machines;76
11.4.1.3.1;3.3.1.3.1 Properties of VMs;76
11.4.1.3.2;3.3.1.3.2 Operations of VMs;77
11.4.1.3.3;3.3.1.3.3 States of VMs;78
11.4.1.3.4;3.3.1.3.4 Typical configurations of VMs;78
11.4.1.4;3.3.1.4 Virtual clusters;78
11.4.1.4.1;3.3.1.4.1 Main properties of a virtual cluster;78
11.4.1.4.2;3.3.1.4.2 States of a virtual cluster;78
11.4.1.4.3;3.3.1.4.3 Operations of a virtual cluster;79
11.4.1.4.4;3.3.1.4.4 Operational errors on VMs;80
11.4.1.5;3.3.1.5 Schedule domains;80
11.4.1.5.1;3.3.1.5.1 Properties of schedule domains;80
11.4.1.5.2;3.3.1.5.2 Operations of schedule domains;81
11.4.1.5.3;3.3.1.5.3 States of schedule domains;81
11.4.1.6;3.3.1.6 Storage;81
11.4.1.6.1;3.3.1.6.1 Properties of shared storage;82
11.4.1.6.2;3.3.1.6.2 Storage operations;83
11.4.1.6.3;3.3.1.6.3 States of storage;83
11.4.1.7;3.3.1.7 VM image;83
11.4.1.7.1;3.3.1.7.1 Properties of a VM image;83
11.4.1.7.2;3.3.1.7.2 VM image operations;84
11.4.1.7.3;3.3.1.7.3 States of VMs;84
11.4.1.8;3.3.1.8 Network resources;84
11.4.1.8.1;3.3.1.8.1 Network resource properties;85
11.4.1.9;3.3.1.9 Data centers;86
11.4.1.9.1;3.3.1.9.1 Properties of data centers;86
11.4.1.9.2;3.3.1.9.2 Data center states;86
11.4.1.9.3;3.3.1.9.3 Data center operations;87
11.4.1.10;3.3.1.10 Machine room resources;87
11.4.1.10.1;3.3.1.10.1 Space;87
11.4.1.10.2;3.3.1.10.2 Power supply;87
11.4.1.10.3;3.3.1.10.3 Air conditioning;88
11.5;3.4 Constraints and dependencies among resources;88
11.5.1;3.4.1 Software/hardware based relations;88
11.5.2;3.4.2 Associated hardware and software platforms and network;88
11.5.3;3.4.3 Reliability constraints;88
11.5.4;3.4.4 Time constraints;89
11.5.5;3.4.5 Relationship among performance, system capacity (storage), and bandwidth;89
11.5.6;3.4.6 Scheduling domain constraints on the scope of algorithm execution;89
11.6;3.5 Data modeling of resources in a Cloud data center;89
11.6.1;3.5.1 Relationship of resources;89
11.6.2;3.5.2 Data management of main resource;89
11.6.2.1;3.5.2.1 Data center;89
11.6.2.2;3.5.2.2 Schedule domains;90
11.6.3;3.5.3 Physical machine queries;91
11.6.4;3.5.4 Add physical machine;91
11.6.5;3.5.5 Delete physical machine;92
11.6.6;3.5.6 Update physical machine information;92
11.6.7;3.5.7 Query VM;92
11.6.8;3.5.8 Add VM;93
11.6.9;3.5.9 Delete VM;94
11.6.10;3.5.10 Update VM information;94
11.7;3.6 Conclusion;94
11.8;Appendix 1: The UML Relationship of Resources;95
11.9;References;96
12;4 Cloud Resource Scheduling Strategies;98
12.1;Main Contents of this Chapter;98
12.2;4.1 Key technologies of resource scheduling;98
12.3;4.2 Comparative analysis of scheduling strategies;99
12.3.1;4.2.1 Amazon;99
12.3.2;4.2.2 IBM;99
12.3.2.1;4.2.2.1 Performance related: satisfying user requirements;99
12.3.3;4.2.3 HP;100
12.3.3.1;4.2.3.1 Cost based: cost model;100
12.3.3.2;4.2.3.2 Load balance: automatically assesses virtual machine burden and carries out dynamic migration;101
12.3.4;4.2.4 VMWare;101
12.3.4.1;4.2.4.1 Improve resource utilization;101
12.3.4.2;4.2.4.2 Improve reliability;101
12.3.4.3;4.2.4.3 Load balance: Distributed Resource Scheduling;102
12.3.5;4.2.5 Other solutions;103
12.3.5.1;4.2.5.1 Fair scheduling;103
12.3.5.2;4.2.5.2 Load balance;103
12.3.5.3;4.2.5.3 Delayed scheduling for locality in Hadoop;103
12.3.5.4;4.2.5.4 Reliability improvement;103
12.4;4.3 Classification of main scheduling strategies;104
12.4.1;4.3.1 Performance related;104
12.4.1.1;4.3.1.1 First come, first served;104
12.4.1.2;4.3.1.2 Load balance;104
12.4.1.3;4.3.1.3 Improve reliability;104
12.4.2;4.3.2 Cost based;105
12.4.2.1;4.3.2.1 Improve overall utilization;105
12.4.2.2;4.3.2.2 Maximum profit;105
12.4.2.3;4.3.2.3 Minimum operation costs;106
12.4.2.4;4.3.2.4 Combining scheduling strategies;106
12.5;4.4 Some constraints of scheduling strategies;109
12.5.1;4.4.1 Space: association and anti-association;109
12.5.2;4.4.2 Scheduling domain: scheduling locality;109
12.5.3;4.4.3 Time: limited available time;109
12.5.4;4.4.4 Migration versus nonmigration;109
12.6;4.5 Scheduling task execution time and trigger conditions;109
12.7;Summary;110
12.8;Appendix: Some elementary terms;110
12.8.1;Resource;110
12.8.2;Resource provider;110
12.8.3;Resource user;110
12.8.4;Resource scheduling;111
12.8.5;Resource management;111
12.8.6;Scheduling strategy;111
12.9;References;111
13;5 Load Balance Scheduling for Cloud Data Centers;114
13.1;Main Contents of this Chapter;114
13.2;5.1 Introduction;114
13.3;5.2 Related work;115
13.4;5.3 Problem formulation and description;115
13.4.1;5.3.1 Metrics for real-time load balancing scheduling algorithms;118
13.5;5.4 OLRSA algorithm;120
13.5.1;5.4.1 Algorithm description;120
13.5.2;5.4.2 Mythology and simulation settings;123
13.5.3;5.4.3 Simulation results and analysis for OLRSA;124
13.5.3.1;5.4.3.1 Divisible capacity configuration of VMs and PMs;124
13.6;5.5 LIF algorithm;125
13.6.1;5.5.1 Description of LIF algorithms;126
13.6.2;5.5.2 Simulation results;128
13.7;5.6 Discussion and conclusion;132
13.8;References;132
14;6 Energy-efficient Allocation of Real-time Virtual Machines in Cloud Data Centers Using Interval-packing Techniques;134
14.1;Main Contents of this Chapter;134
14.2;6.1 Introduction;134
14.3;6.2 GreenCloud architecture;136
14.3.1;6.2.1 Cloud data center resources;137
14.3.2;6.2.2 A uniform view for different types of VMs;137
14.3.3;6.2.3 Real-time VM request model;138
14.4;6.3 Energy-efficient real-time scheduling;139
14.4.1;6.3.1 Problem description;139
14.4.1.1;6.3.1.1 The linear power consumption model of a server;140
14.4.1.2;6.3.1.2 Capacity configuration of VMs and PMs;142
14.4.1.2.1;6.3.1.2.1 Random capacity configuration of VMs and PMs;142
14.4.1.2.2;6.3.1.2.2 Divisible capacity configuration of VMs and PMs;142
14.4.1.3;6.3.1.3 The power usage policies;143
14.4.1.3.1;6.3.1.3.1 Strategy one: idle servers turned off;143
14.4.1.3.2;6.3.1.3.2 Strategy two: idle servers not turned off;144
14.4.2;6.3.2 Four offline and online scheduling algorithms;145
14.5;6.4 Performance evaluation;146
14.5.1;6.4.1 Methodology;146
14.5.2;6.4.2 Metrics;147
14.5.3;6.4.3 Algorithms;147
14.5.4;6.4.4 Inputs settings and results analysis;147
14.5.4.1;6.4.4.1 Assuming idle servers turned off;148
14.5.4.2;6.4.4.2 Assuming idle servers not turned off;150
14.5.4.3;6.4.4.3 Impact of varying the total number of VM requests;150
14.6;6.5 Related work;151
14.7;6.6 Conclusions;151
14.8;References;151
15;7 Energy Efficiency by Minimizing Total Busy Time of Offline Parallel Scheduling in Cloud Computing;154
15.1;Main Contents of this Chapter;154
15.2;7.1 Introduction;154
15.2.1;7.1.1 Related work;155
15.2.2;7.1.2 Preliminaries;156
15.2.3;7.1.3 Results;158
15.3;7.2 Approximation algorithm and its approximation ratio bound;159
15.3.1;7.2.1 Bounds for approximation ratio when g is one unit and di is one unit;160
15.3.2;7.2.2 Bounds for the approximation ratio in the general case when g>1;160
15.4;7.3 Application to energy efficiency in Cloud computing;165
15.4.1;7.3.1 Problem formulation;165
15.4.2;7.3.2 Average case analysis;168
15.5;7.4 Performance evaluation;168
15.5.1;7.4.1 Methodology;168
15.5.2;7.4.2 Algorithms;169
15.5.3;7.4.3 Simulation using real traces;169
15.5.4;7.4.4 Simulation using synthetic data;170
15.5.4.1;7.4.4.1 Data center energy consumption evaluation;170
15.5.4.2;7.4.4.2 Impact of total workload;172
15.5.5;7.4.5 General observations;174
15.6;7.5 Conclusions;174
15.7;References;175
16;8 Comparative Study of Energy-efficient Scheduling in Cloud Data Centers;178
16.1;Main Contents of this Chapter;178
16.2;8.1 Introduction;178
16.3;8.2 Related research;180
16.4;8.3 Comparative study of offline scheduling algorithms;181
16.4.1;8.3.1 Energy models for data centers;181
16.4.1.1;8.3.1.1 Data center energy consumption evaluation;181
16.4.1.2;8.3.1.2 Server power consumption model;182
16.4.2;8.3.2 FFD algorithm;183
16.4.3;8.3.3 MFFDE algorithm;184
16.4.4;8.3.4 Other offline algorithms;184
16.4.4.1;8.3.4.1 The STF algorithm;184
16.4.4.2;8.3.4.2 The earliest ending-time first algorithm;184
16.4.4.3;8.3.4.3 Random allocation algorithm (Random);186
16.5;8.4 Online algorithms;186
16.5.1;8.4.1 BFF algorithm;186
16.5.1.1;8.4.1.1 The bounds for the competitive ratio when g is one unit and di is one unit;188
16.5.1.2;8.4.1.2 The bounds for the competitive ratio in the general case when g>1;188
16.5.2;8.4.2 GRID algorithm;192
16.5.2.1;8.4.2.1 GREEDYBUCKET is g-competitive in the worst-case scenario;192
16.6;8.5 Summary;196
16.7;References;196
17;9 Energy Efficiency Scheduling in Hadoop;198
17.1;Main Contents of this Chapter;198
17.2;9.1 Overview;198
17.2.1;9.1.1 Hadoop introduction;198
17.2.2;9.1.2 Hadoop framework;198
17.2.3;9.1.3 Hadoop running processes;200
17.3;9.2 Scheduling algorithms;201
17.3.1;9.2.1 Dynamic management of Hadoop clusters;201
17.3.2;9.2.2 Load modeling;203
17.3.2.1;9.2.2.1 Load information;203
17.3.2.2;9.2.2.2 Period;203
17.3.2.3;9.2.2.3 Negative feedback mechanism;203
17.3.3;9.2.3 Scheduling algorithm;204
17.3.3.1;9.2.3.1 Scheduling conditions;204
17.3.3.2;9.2.3.2 Choosing a node to suspend;204
17.3.3.3;9.2.3.3 Choosing a node to restart;205
17.3.3.4;9.2.3.4 Pseudocode;205
17.4;9.3 Energy control;205
17.4.1;9.3.1 System architecture;205
17.4.2;9.3.2 Detailed design;206
17.4.2.1;9.3.2.1 Resource collection;206
17.4.2.2;9.3.2.2 Remote control;206
17.4.2.3;9.3.2.3 Node control;206
17.5;9.4 Energy-efficient scheduling for multiple users;207
17.5.1;9.4.1 Problem formulation;207
17.5.2;9.4.2 Revised Johnson’s algorithm and HScheduler;209
17.5.2.1;9.4.2.1 Johnson’s algorithm revisited;209
17.6;9.5 Performance evaluation;214
17.6.1;9.5.1 Evaluation platform;214
17.6.2;9.5.2 Evaluation design;214
17.6.3;9.5.3 Results analysis;214
17.6.3.1;9.5.3.1 Energy control system;214
17.6.3.2;9.5.3.2 Energy-efficient scheduling;219
17.7;9.6 Summary;221
17.8;Questions;222
17.9;References;222
18;10 Maximizing Total Weights in Virtual Machines Allocation;224
18.1;Main Contents of this Chapter;224
18.2;10.1 Introduction;224
18.3;10.2 Problem formulation: WISWCS;225
18.3.1;10.2.1 Traditional WISP;225
18.4;10.3 WISWCS;228
18.5;10.4 An exact SAWISWCS;230
18.6;10.5 Applications of WISWCS;232
18.6.1;10.5.1 Virtual machine scheduling in cloud computing;233
18.6.2;10.5.2 Performance evaluation;233
18.7;10.6 Related work;234
18.8;10.7 Conclusions;234
18.9;References;234
19;11 A Toolkit for Modeling and Simulation of Real-time Virtual Machine Allocation in a Cloud Data Center;236
19.1;Main Contents of this Chapter;236
19.2;11.1 Introduction of the cloud data center;236
19.3;11.2 The architecture and main features of CloudSched;239
19.3.1;11.2.1 Modeling CDCs;241
19.3.2;11.2.2 Modeling VM allocation;242
19.3.3;11.2.3 Modeling customer requirements;243
19.4;11.3 Performance metrics for different scheduling algorithms;244
19.4.1;11.3.1 Metrics for multidimensional load balancing;244
19.4.2;11.3.2 Metrics for energy efficiency;247
19.4.2.1;11.3.2.1 Power consumption model;247
19.4.3;11.3.3 Metric for maximizing resource utilization;247
19.5;11.4 Design and implementation of CloudSched;248
19.5.1;11.4.1 IaaS resources considered;248
19.5.2;11.4.2 Scheduling process in CDC;248
19.5.3;11.4.3 Scheduling algorithms: taking the LIF algorithm as an example;248
19.6;11.5 Performance evaluation;253
19.6.1;11.5.1 Random configuration of VMs and PMs;253
19.6.2;11.5.2 Divisible size configuration of PMs and VMs;257
19.6.3;11.5.3 Comparing energy efficiency;257
19.6.3.1;11.5.3.1 Impact of varying maximum duration of VM requests;259
19.6.3.2;11.5.3.2 Impact of varying the total number of VM requests;259
19.7;11.6 Conclusions;259
19.8;References;261
20;12 Toward Running Scientific Workflows in the Cloud;264
20.1;Main Contents of this Chapter;264
20.2;12.1 Introduction;264
20.3;12.2 Related work;266
20.4;12.3 Integration;267
20.4.1;12.3.1 Integration options;267
20.4.1.1;12.3.1.1 Operational-Layer-in-the-cloud;268
20.4.1.2;12.3.1.2 Task-Management-Layer-in-the-cloud;268
20.4.1.3;12.3.1.3 Workflow-Management-Layer-in-the-cloud;269
20.4.1.4;12.3.1.4 All-in-the-cloud;269
20.4.2;12.3.2 The Swift workflow management system;269
20.4.3;12.3.3 Integration challenges;270
20.4.4;12.3.4 Integration architecture;271
20.4.4.1;12.3.4.1 The client submission tool;271
20.4.4.2;12.3.4.2 The Swift Cloud workflow service;272
20.4.4.3;12.3.4.3 The CRM;272
20.5;12.4 Experiment;273
20.5.1;12.4.1 MODIS image processing workflow;273
20.5.2;12.4.2 Experiment configuration;273
20.5.3;12.4.3 Experiment results;274
20.5.3.1;12.4.3.1 The serial submission experiment;274
20.5.3.2;12.4.3.2 The parallel submission experiment;276
20.5.3.3;12.4.3.3 Different number of data blocks experiment;277
20.6;12.5 Experiment on Amazon EC2;278
20.6.1;12.5.1 Montage image processing workflow;278
20.6.2;12.5.2 Experiment configuration;279
20.6.3;12.5.3 Experiment results;279
20.6.3.1;12.5.3.1 Falkon cluster initialization experiment;279
20.6.3.2;12.5.3.2 MASS nebula graph processing experiment;282
20.7;12.6 Conclusions;283
20.8;References;284