Deelman / Shields Workflows for e-Science

Scientific Workflows for Grids
1. Auflage 2007
ISBN: 978-1-84628-757-2
Verlag: Springer-Verlag
Format: PDF
Kopierschutz: Adobe DRM (»Systemvoraussetzungen)

E-Book, Englisch, 526 Seiten

This collection of articles on ‘Work?ows for e-Science’ is very timely and - portant. Increasingly, to attack the next generation of scienti?c problems, multidisciplinary and distributed teams of scientists need to collaborate to make progress on these new ‘Grand Challenges’. Scientists now need to access and exploit computational resources and databases that are geographically distributed through theuseof high speed networks. ‘Virtual Organizations’ or ‘VOs’ must be established that span multiple administrative domains and/or institutions and which can provide appropriate authentication and author- ation services and access controls to collaborating members. Some of these VOsmayonlyhavea?eetingexistencebutthelifetimeofothersmayrun into many years. The Grid community is attempting to develop both sta- ards and middleware to enable both scientists and industry to build such VOs routinely and robustly. This, of course, has been the goal of research in distributed computing for many years; but now these technologies come with a new twist service orie- ation. By specifying resources in terms of a service description, rather than allowing direct access to the resources, the IT industry believes that such an approach results in the construction of more robust distributed systems. The industry has therefore united around web services as the standard technology toimplementsuchserviceorientedarchitecturesandtoensureinteroperability between di?erent vendor systems.

Autoren/Hrsg.

Deelman, Ewa

Shields, Matthew

Inhaltsverzeichnis

1;Foreword;7
2;Contents;9
3;List of Contributors;13
4;1 Introduction;22
4.1;1.1 Background;22
4.2;1.2 Application and User Perspective;24
4.3;1.3 Work.ow Representation and Common Structure;25
4.4;1.4 Frameworks and Tools: Work.ow Generation, Refinement and Execution;26
5;2 Scientific versus Business Workflows;30
6;Part I Application and User Perspective;38
6.1;3 Generating Complex Astronomy Workflows;40
6.1.1;3.1 Introduction;40
6.1.2;3.2 The Architecture of Montage;41
6.1.3;3.3 Grid-Enabled Montage;46
6.1.4;3.4 Supporting a Community of Users;52
6.1.5;Acknowledgments;58
6.2;4 A Case Study on the Use of Workflow Technologies for Scientific Analysis: Gravitational Wave Data Analysis;60
6.2.1;4.1 Introduction;60
6.2.2;4.2 Gravitational Waves;60
6.2.3;4.3 The LIGO Data Grid Infrastructure;63
6.2.4;4.4 Constructing Workflows with the Grid/LSC User Environment;68
6.2.5;4.5 The Inspiral Analysis Workflow;73
6.2.6;4.6 Concluding Remarks;79
6.2.7;Acknowledgments;80
6.3;5 Workflows in Pulsar Astronomy;81
6.3.1;5.1 Introduction;81
6.3.2;5.2 Pulsars and Their Detection;81
6.3.3;5.3 Workflow for Signal Processing;83
6.3.4;5.4 Use of Metacomputing in Dedispersion;88
6.3.5;5.5 Workflows of Online Pulsar Searches;92
6.3.6;5.6 Future Work: Toward a Service-Oriented Approach;97
6.3.7;Acknowledgments;99
6.4;6 Workflow and Biodiversity e-Science;101
6.4.1;6.1 Introduction;101
6.4.2;6.2 Background: Biodiversity and e-Science;101
6.4.3;6.3 BiodiversityWorld as an e-Biodiversity Environment;103
6.4.4;6.4 Related Work;107
6.4.5;6.5 Toward an Exploratory Workflow Environment;109
6.4.6;6.6 Conclusions;110
6.4.7;Acknowledgments;111
6.5;7 Ecological Niche Modeling Using the Kepler Workflow System;112
6.5.1;7.1 Introduction;112
6.5.2;7.2 Approaches in Ecological Niche Modeling;113
6.5.3;7.3 Data Access via EcoGrid;116
6.5.4;7.4 Hierarchical Decomposition of the ENM Workflow;116
6.5.5;7.5 Modular Component Substitution;119
6.5.6;7.6 Transformation and Data Integration;122
6.5.7;7.7 Grid and Peer-to-Peer Computing;125
6.5.8;7.8 Opportunities for Biodiversity Science Using Scientific Workflows;126
6.5.9;7.9 Advantages of Automated Workflows for Biodiversity and Ecological Science;128
6.5.10;Acknowledgments;129
6.6;8 Case Studies on the Use of Workflow Technologies for Scientific Analysis: The Biomedical Informatics Research Network and the Telescience Project;130
6.6.1;8.1 Introduction;130
6.6.2;8.2 Framework for Integrated Workflow Environments;132
6.6.3;8.3 Scientific Process Workflows: Process and State Management Tools;135
6.6.4;8.4 The Role of Portals as Workflow Controllers;136
6.6.5;8.5 Interapplication Workflows: Pipeline-Building Tools;137
6.6.6;8.6 Intrapipeline Workflow: Planners and Execution Engines;139
6.6.7;8.7 Use Cases;140
6.6.8;8.8 The Telescience Project;140
6.6.9;8.9 The Biomedical Informatics Research Network ( BIRN);141
6.6.10;8.10 Discussion;146
6.6.11;Acknowledgments;146
6.7;9 Dynamic, Adaptive Workflows for Mesoscale Meteorology;147
6.7.1;9.1 Introduction;147
6.7.2;9.2 The LEAD Data and Service Architecture;149
6.7.3;9.3 LEAD Workflow;151
6.7.4;9.4 Conclusions;162
6.7.5;9.5 Acknowledgments;163
6.8;10 SCEC CyberShake Workflows—Automating Probabilistic Seismic Hazard Analysis Calculations;164
6.8.1;10.1 Introduction to SCEC CyberShake Workflows;164
6.8.2;10.2 The SCEC Hardware and Software Computing Environment;167
6.8.3;10.3 SCEC Probabilistic Seismic Hazard Analysis Research;168
6.8.4;10.4 Computational Requirements of CyberShake;169
6.8.5;10.5 SCEC Work.ow Solutions to Key Workflow Requirements;172
6.8.6;10.6 Benefits of Modeling CyberShake as Workflows;173
6.8.7;10.7 Cost of Using the SCEC Workflow System;174
6.8.8;10.8 From Computational Pathway to Abstract Workflow;175
6.8.9;10.9 Resource Provisioning in the CyberShake Workflows;181
6.8.10;10.10 CyberShake Workflow Results;182
6.8.11;10.11 Conclusions;183
6.8.12;Acknowledgments;184
7;Part II Workflow Representation and Common Structure;186
7.1;11 Control- Versus Data-Driven Workflows;188
7.1.1;11.1 Introduction;188
7.1.2;11.2 Workflow Representations;189
7.1.3;11.3 Control-Driven Workflows;191
7.1.4;11.4 Data-Driven Workflows;193
7.1.5;11.5 Toward a Common Workflow Language;193
7.2;12 Component Architectures and Services: From Application Construction to Scientific Workflows;195
7.2.1;12.1 Introduction;195
7.2.2;12.2 Component Architectures: General Concepts;196
7.2.3;12.3 Models of Composition;199
7.2.4;12.4 Stateful and Stateless Components;206
7.2.5;12.5 Space and Time and the Limits to the Power of Graphical Expression;208
7.3;13 Petri Nets;211
7.3.1;13.1 Introduction;211
7.3.2;13.2 Choreography—Using Petri Nets for Modelling Abstract Applications;215
7.3.3;13.3 Orchestration—Using Petri Nets for Mapping Abstract Workflows onto Concrete Resources;222
7.3.4;13.4 Enactment—Using Petri Nets for Executing and Controlling e- Science Applications;223
7.3.5;13.5 Conclusions;227
7.3.6;Acknowledgments;228
7.4;14 Adapting BPEL to Scientific Workflows;229
7.4.1;14.1 Introduction;229
7.4.2;14.2 Short Overview of BPEL;229
7.4.3;14.3 Goals and Requirements for Scientific Workflows in Grids;234
7.4.4;14.4 Illustrative Grid Workflow Example;236
7.4.5;14.5 Workflow Life-Cycle on an Example of a GPEL Engine;240
7.4.6;14.6 Challenges in Using BPEL in Grids;246
7.5;15 Protocol-Based Integration Using SSDL and p-Calculus;248
7.5.1;15.1 Introduction;248
7.5.2;15.2 Service Orientation;250
7.5.3;15.3 SSDL Overview;252
7.5.4;15.4 The Sequential Constraint Protocol Framework;255
7.5.5;15.5 A Use Case;259
7.5.6;15.6 Related Work;262
7.5.7;15.7 Conclusions;264
7.5.8;Acknowledgments;264
7.6;16 Workflow Composition: Semantic Representations for Flexible Automation;265
7.6.1;16.1 Introduction;265
7.6.2;16.2 The Need for Assisted Workflow Composition;265
7.6.3;16.3 From Reusable Templates to Fully Specified Executable Workflows;271
7.6.4;16.4 Semantic Representations of Workflows to Support Assisted Composition;275
7.6.5;16.5 Automatic Completion of Workflows;277
7.6.6;16.6 Conclusions;278
7.6.7;Acknowledgments;278
7.7;17 Virtual Data Language: A Typed Workflow Notation for Diversely Structured Scientific Data;279
7.7.1;17.1 Introduction;279
7.7.2;17.2 Related Work;281
7.7.3;17.3 XDTM Overview;282
7.7.4;17.4 Physical and Logical Structure: An Example;282
7.7.5;17.5 Virtual Data Language;283
7.7.6;17.6 An Application Example: Functional MRI;290
7.7.7;17.7 VDL Implementation;294
7.7.8;17.8 Conclusion;296
7.7.9;Acknowledgments;296
8;Part III Frameworks and Tools: Work.ow Generation, Re . nement, and Execution;298
8.1;18 Workflow-Level Parametric Study Support by MOTEUR and the P-GRADE Portal;300
8.1.1;18.1 Introduction;300
8.1.2;18.2 Task-Based and Service-Based Workflows;301
8.1.3;18.3 Describing Parametric Application Workflows;302
8.1.4;18.4 Efficient Execution of Data-Intensive Workflows;304
8.1.5;18.5 Exploiting Both Task- and Service-Based Approaches in Parametric Data- Intensive Applications;311
8.1.6;18.6 MOTEUR Service-Based Work.ow Enactor;312
8.1.7;18.7 P-GRADE Portal;313
8.1.8;18.8 Conclusions;319
8.1.9;18.9 Acknowledgments;320
8.2;19 Taverna/myGrid: Aligning a Workflow System with the Life Sciences Community;321
8.2.1;19.1 Introduction;321
8.2.2;19.2 The Bioinformatics Background;324
8.2.3;19.3 Aligning with Life Science;325
8.2.4;19.4 Architecture of Taverna;326
8.2.5;19.5 Discovering Resources and Designing Workflows;331
8.2.6;19.6 Executing and Monitoring Workflows;334
8.2.7;19.7 Managing and Sharing Workflows and Their Results;336
8.2.8;19.8 Related Work;337
8.2.9;19.9 Discussion and Future Directions;339
8.2.10;Acknowledgments;340
8.3;20 The Triana Workflow Environment: Architecture and Applications;341
8.3.1;20.1 Introduction;341
8.3.2;20.2 Relation to Other Frameworks;343
8.3.3;20.3 Inside The Triana Framework;344
8.3.4;20.4 Distributed Triana Workflows;345
8.3.5;20.5 Workflow Representation and Generation;351
8.3.6;20.6 Current Triana Applications;353
8.3.7;20.7 Example 1: Distributing GAP Services;354
8.3.8;20.8 Example 2: The Visual GAT;356
8.3.9;20.9 Conclusion;360
8.3.10;20.10 Acknowledgments;360
8.4;21 Java CoG Kit Workflow;361
8.4.1;21.1 Introduction;361
8.4.2;21.2 The Java CoG Kit Karajan Workflow Framework;366
8.4.3;21.3 Work.ow Support for Experiment Management;376
8.4.4;21.4 Conclusion;376
8.4.5;Acknowledgement;377
8.5;22 Workflow Management in Condor;378
8.5.1;22.1 Introduction;378
8.5.2;22.2 DAGMan Design Principles;379
8.5.3;22.3 DAGMan Details;380
8.5.4;22.4 Implementation Status;389
8.5.5;22.5 Interaction with Condor;390
8.5.6;22.6 Integration with Stork;390
8.5.7;22.7 Future Directions;395
8.5.8;22.8 Conclusions;396
8.6;23 Pegasus: Mapping Large-Scale Work.ows to Distributed Resources;397
8.6.1;23.1 Introduction;397
8.6.2;23.2 Workflow Generation for Pegasus;398
8.6.3;23.3 Pegasus and the Target Workflow Execution Environment;399
8.6.4;23.4 Pegasus and Workflow Refinement;402
8.6.5;23.5 Workflow Execution;406
8.6.6;23.6 Adapting the Workflow Mapping to a Dynamic Execution Environment;406
8.6.7;23.7 Optimizing Workflow Performance with Pegasus;408
8.6.8;23.8 Applications;411
8.6.9;23.9 Related Work;413
8.6.10;23.10 Conclusions;414
8.6.11;Acknowledgements;415
8.7;24 ICENI;416
8.7.1;24.1 Introduction;416
8.7.2;24.2 The Workflow Pipeline;423
8.7.3;24.3 Specification;424
8.7.4;24.4 Realization;426
8.7.5;24.5 Execution Environment;431
8.7.6;24.6 Application Interaction;435
8.7.7;24.7 Conclusion;435
8.8;25 Expressing Workflow in the Cactus Framework;437
8.8.1;25.1 Introduction;437
8.8.2;25.2 Structure;438
8.8.3;25.3 Basic Workflow in Cactus;438
8.8.4;25.4 Extensions;442
8.9;26 Sedna: A BPEL-Based Environment for Visual Scientific Workflow Modeling;449
8.9.1;26.1 Introduction;449
8.9.2;26.2 Modeling Scientific Workflows;451
8.9.3;26.3 Scientific Workflow Editor;457
8.9.4;26.4 Case Study: Polymorph Search;465
8.9.5;26.5 Related Work;468
8.9.6;26.6 Lessons Learned and Future Work;469
8.9.7;26.7 Acknowledgments;470
8.10;27 ASKALON: A Development and Grid Computing Environment for Scienti fic Workflows;471
8.10.1;27.1 Introduction;471
8.10.2;27.2 Work.ow Case Study and Grid Infrastructure;472
8.10.3;27.3 Work.ow Generation;474
8.10.4;27.4 Resource Manager;477
8.10.5;27.5 Scheduler;479
8.10.6;27.6 Execution Engine;484
8.10.7;27.7 Overhead Analysis;486
8.10.8;27.8 Conclusions;491
8.10.9;27.9 Acknowledgments;492
9;Part IV Future Requirements;494
9.1;Looking into the Future of Workflows: The Challenges Ahead;496
9.1.1;1 User Experience;496
9.1.2;2 Workflow Languages and Representations;498
9.1.3;3 Workflow Compilers;499
9.1.4;4 Workflow Enactors or Executors;500
9.1.5;5 Debugging;501
9.1.6;6 Execution Environments;501
9.1.7;7 The Big Question;502
10;References;504
11;Index;536