Taylor / Deelman / Gannon Workflows for e-Science

Scientific Workflows for Grids
1. Auflage 2007
ISBN: 978-1-84628-757-2
Verlag: Springer
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 526 Seiten

This is a timely book presenting an overview of the current state-of-the-art within established projects, presenting many different aspects of workflow from users to tool builders. It provides an overview of active research, from a number of different perspectives. It includes theoretical aspects of workflow and deals with workflow for e-Science as opposed to e-Commerce. The topics covered will be of interest to a wide range of practitioners.

Dr. Ian Taylor has been a Lecturer at Cardiff University's School of Computer Science since 2002. He concurrently holds an adjunct Assistant Professorship at the Center for Computation & Technology at Louisiana State University and regularly offers consultations in the USA. He has a Ph.D. in Physics and Music and is the co-ordinator of Triana activities at Cardiff (http://www.trianacode.org). Through this he has been active in many major projects including GridLab, CoreGrid and GridOneD. His research interests include distributed techniques and workflow for Grid and P2P computing, which take in applications ranging from astrophysics and healthcare to distributed audio. Ian has previously written a professional book for Springer on P2P, Web Services and Grids, and has published over 50 scientific papers. He has also co-edited a special edition for Journal of Grid Computing on Scientific Workflow. Dr. Matthew Shields has been a research associate at Cardiff University, jointly in the Schools of Computer Science,
Physics and Astronomy, since 2001. He gained his Ph.D. in Computer Science from Cardiff University in the area of
problem solving environments. Dr Shields is one of two lead developers for the Triana project and has been responsible
for helping broaden its adoption within new application domains including biodiversity. His interests include problem
solving environments, workflow, component and service based computing, Grid and high-performance computing. Ewa Deelman is an Research Assistant Professor at the USC Computer Science Department and a Research Team Leader at the Center for Grid Technologies at the USC Information Sciences Institute. Dr. Deelman's research interests include the design and exploration of collaborative scientific environments based on Grid technologies, with particular emphasis on workflow management as well as the management of large amounts of data and metadata. At ISI, Dr. Deelman is leading the Pegasus project, which designs and implements workflow mapping techniques for large-scale workflows running in distributed environments. Pegasus is being used day-to-day by scientists in a variety of disciplines including astronomy, gravitational-wave physics, earthquake science and many others. Prior to joining ISI in 2000, she was a Senior Software Developer at UCLA conducting research in the area of performance prediction of large-scale applications on high performance machines. Dr. Deelman received her PhD from Rensselaer Polytechnic Institute in Computer Science in 1997 in the area of parallel discrete event simulation. Dr. Deelman is an Associate Editor responsible for Grid Computing for the Scientific Programming Journal and a chair of the GGF Workflow Management Research Group. Dennis Gannon, Department of Computer Science, Lindley Hall, Indiana University, Bloomington, IN 47401 ([e-mail]) Dr. Gannon is a professor of Computer Science in the School of Informatics at Indiana University. He is also Science Director for the Indiana Pervasive Technology Labs. He received his Ph.D. in Computer Science from the University of Illinois in 1980 and his Ph.D. in Mathematics from the University of California in 1974. From 1980 to 1985, he was on the faculty at Purdue University. From 1997-2004 he was Chairman of the Indiana Computer Science Department. His research interests include software tools for high performance parallel and distributed systems and problem solving environments for scientific computation.

Autoren/Hrsg.

Deelman, Ewa

Shields, Matthew

Taylor, Ian J.

Gannon, Dennis B.

Inhaltsverzeichnis

1;Foreword;7
2;Contents;9
3;List of Contributors;13
4;1 Introduction;22
4.1;1.1 Background;22
4.2;1.2 Application and User Perspective;24
4.3;1.3 Work.ow Representation and Common Structure;25
4.4;1.4 Frameworks and Tools: Work.ow Generation, Refinement and Execution;26
5;2 Scientific versus Business Workflows;30
6;Part I Application and User Perspective;38
6.1;3 Generating Complex Astronomy Workflows;40
6.1.1;3.1 Introduction;40
6.1.2;3.2 The Architecture of Montage;41
6.1.3;3.3 Grid-Enabled Montage;46
6.1.4;3.4 Supporting a Community of Users;52
6.1.5;Acknowledgments;58
6.2;4 A Case Study on the Use of Workflow Technologies for Scientific Analysis: Gravitational Wave Data Analysis;60
6.2.1;4.1 Introduction;60
6.2.2;4.2 Gravitational Waves;60
6.2.3;4.3 The LIGO Data Grid Infrastructure;63
6.2.4;4.4 Constructing Workflows with the Grid/LSC User Environment;68
6.2.5;4.5 The Inspiral Analysis Workflow;73
6.2.6;4.6 Concluding Remarks;79
6.2.7;Acknowledgments;80
6.3;5 Workflows in Pulsar Astronomy;81
6.3.1;5.1 Introduction;81
6.3.2;5.2 Pulsars and Their Detection;81
6.3.3;5.3 Workflow for Signal Processing;83
6.3.4;5.4 Use of Metacomputing in Dedispersion;88
6.3.5;5.5 Workflows of Online Pulsar Searches;92
6.3.6;5.6 Future Work: Toward a Service-Oriented Approach;97
6.3.7;Acknowledgments;99
6.4;6 Workflow and Biodiversity e-Science;101
6.4.1;6.1 Introduction;101
6.4.2;6.2 Background: Biodiversity and e-Science;101
6.4.3;6.3 BiodiversityWorld as an e-Biodiversity Environment;103
6.4.4;6.4 Related Work;107
6.4.5;6.5 Toward an Exploratory Workflow Environment;109
6.4.6;6.6 Conclusions;110
6.4.7;Acknowledgments;111
6.5;7 Ecological Niche Modeling Using the Kepler Workflow System;112
6.5.1;7.1 Introduction;112
6.5.2;7.2 Approaches in Ecological Niche Modeling;113
6.5.3;7.3 Data Access via EcoGrid;116
6.5.4;7.4 Hierarchical Decomposition of the ENM Workflow;116
6.5.5;7.5 Modular Component Substitution;119
6.5.6;7.6 Transformation and Data Integration;122
6.5.7;7.7 Grid and Peer-to-Peer Computing;125
6.5.8;7.8 Opportunities for Biodiversity Science Using Scientific Workflows;126
6.5.9;7.9 Advantages of Automated Workflows for Biodiversity and Ecological Science;128
6.5.10;Acknowledgments;129
6.6;8 Case Studies on the Use of Workflow Technologies for Scientific Analysis: The Biomedical Informatics Research Network and the Telescience Project;130
6.6.1;8.1 Introduction;130
6.6.2;8.2 Framework for Integrated Workflow Environments;132
6.6.3;8.3 Scientific Process Workflows: Process and State Management Tools;135
6.6.4;8.4 The Role of Portals as Workflow Controllers;136
6.6.5;8.5 Interapplication Workflows: Pipeline-Building Tools;137
6.6.6;8.6 Intrapipeline Workflow: Planners and Execution Engines;139
6.6.7;8.7 Use Cases;140
6.6.8;8.8 The Telescience Project;140
6.6.9;8.9 The Biomedical Informatics Research Network ( BIRN);141
6.6.10;8.10 Discussion;146
6.6.11;Acknowledgments;146
6.7;9 Dynamic, Adaptive Workflows for Mesoscale Meteorology;147
6.7.1;9.1 Introduction;147
6.7.2;9.2 The LEAD Data and Service Architecture;149
6.7.3;9.3 LEAD Workflow;151
6.7.4;9.4 Conclusions;162
6.7.5;9.5 Acknowledgments;163
6.8;10 SCEC CyberShake Workflows—Automating Probabilistic Seismic Hazard Analysis Calculations;164
6.8.1;10.1 Introduction to SCEC CyberShake Workflows;164
6.8.2;10.2 The SCEC Hardware and Software Computing Environment;167
6.8.3;10.3 SCEC Probabilistic Seismic Hazard Analysis Research;168
6.8.4;10.4 Computational Requirements of CyberShake;169
6.8.5;10.5 SCEC Work.ow Solutions to Key Workflow Requirements;172
6.8.6;10.6 Benefits of Modeling CyberShake as Workflows;173
6.8.7;10.7 Cost of Using the SCEC Workflow System;174
6.8.8;10.8 From Computational Pathway to Abstract Workflow;175
6.8.9;10.9 Resource Provisioning in the CyberShake Workflows;181
6.8.10;10.10 CyberShake Workflow Results;182
6.8.11;10.11 Conclusions;183
6.8.12;Acknowledgments;184
7;Part II Workflow Representation and Common Structure;186
7.1;11 Control- Versus Data-Driven Workflows;188
7.1.1;11.1 Introduction;188
7.1.2;11.2 Workflow Representations;189
7.1.3;11.3 Control-Driven Workflows;191
7.1.4;11.4 Data-Driven Workflows;193
7.1.5;11.5 Toward a Common Workflow Language;193
7.2;12 Component Architectures and Services: From Application Construction to Scientific Workflows;195
7.2.1;12.1 Introduction;195
7.2.2;12.2 Component Architectures: General Concepts;196
7.2.3;12.3 Models of Composition;199
7.2.4;12.4 Stateful and Stateless Components;206
7.2.5;12.5 Space and Time and the Limits to the Power of Graphical Expression;208
7.3;13 Petri Nets;211
7.3.1;13.1 Introduction;211
7.3.2;13.2 Choreography—Using Petri Nets for Modelling Abstract Applications;215
7.3.3;13.3 Orchestration—Using Petri Nets for Mapping Abstract Workflows onto Concrete Resources;222
7.3.4;13.4 Enactment—Using Petri Nets for Executing and Controlling e- Science Applications;223
7.3.5;13.5 Conclusions;227
7.3.6;Acknowledgments;228
7.4;14 Adapting BPEL to Scientific Workflows;229
7.4.1;14.1 Introduction;229
7.4.2;14.2 Short Overview of BPEL;229
7.4.3;14.3 Goals and Requirements for Scientific Workflows in Grids;234
7.4.4;14.4 Illustrative Grid Workflow Example;236
7.4.5;14.5 Workflow Life-Cycle on an Example of a GPEL Engine;240
7.4.6;14.6 Challenges in Using BPEL in Grids;246
7.5;15 Protocol-Based Integration Using SSDL and p-Calculus;248
7.5.1;15.1 Introduction;248
7.5.2;15.2 Service Orientation;250
7.5.3;15.3 SSDL Overview;252
7.5.4;15.4 The Sequential Constraint Protocol Framework;255
7.5.5;15.5 A Use Case;259
7.5.6;15.6 Related Work;262
7.5.7;15.7 Conclusions;264
7.5.8;Acknowledgments;264
7.6;16 Workflow Composition: Semantic Representations for Flexible Automation;265
7.6.1;16.1 Introduction;265
7.6.2;16.2 The Need for Assisted Workflow Composition;265
7.6.3;16.3 From Reusable Templates to Fully Specified Executable Workflows;271
7.6.4;16.4 Semantic Representations of Workflows to Support Assisted Composition;275
7.6.5;16.5 Automatic Completion of Workflows;277
7.6.6;16.6 Conclusions;278
7.6.7;Acknowledgments;278
7.7;17 Virtual Data Language: A Typed Workflow Notation for Diversely Structured Scientific Data;279
7.7.1;17.1 Introduction;279
7.7.2;17.2 Related Work;281
7.7.3;17.3 XDTM Overview;282
7.7.4;17.4 Physical and Logical Structure: An Example;282
7.7.5;17.5 Virtual Data Language;283
7.7.6;17.6 An Application Example: Functional MRI;290
7.7.7;17.7 VDL Implementation;294
7.7.8;17.8 Conclusion;296
7.7.9;Acknowledgments;296
8;Part III Frameworks and Tools: Work.ow Generation, Re . nement, and Execution;298
8.1;18 Workflow-Level Parametric Study Support by MOTEUR and the P-GRADE Portal;300
8.1.1;18.1 Introduction;300
8.1.2;18.2 Task-Based and Service-Based Workflows;301
8.1.3;18.3 Describing Parametric Application Workflows;302
8.1.4;18.4 Efficient Execution of Data-Intensive Workflows;304
8.1.5;18.5 Exploiting Both Task- and Service-Based Approaches in Parametric Data- Intensive Applications;311
8.1.6;18.6 MOTEUR Service-Based Work.ow Enactor;312
8.1.7;18.7 P-GRADE Portal;313
8.1.8;18.8 Conclusions;319
8.1.9;18.9 Acknowledgments;320
8.2;19 Taverna/myGrid: Aligning a Workflow System with the Life Sciences Community;321
8.2.1;19.1 Introduction;321
8.2.2;19.2 The Bioinformatics Background;324
8.2.3;19.3 Aligning with Life Science;325
8.2.4;19.4 Architecture of Taverna;326
8.2.5;19.5 Discovering Resources and Designing Workflows;331
8.2.6;19.6 Executing and Monitoring Workflows;334
8.2.7;19.7 Managing and Sharing Workflows and Their Results;336
8.2.8;19.8 Related Work;337
8.2.9;19.9 Discussion and Future Directions;339
8.2.10;Acknowledgments;340
8.3;20 The Triana Workflow Environment: Architecture and Applications;341
8.3.1;20.1 Introduction;341
8.3.2;20.2 Relation to Other Frameworks;343
8.3.3;20.3 Inside The Triana Framework;344
8.3.4;20.4 Distributed Triana Workflows;345
8.3.5;20.5 Workflow Representation and Generation;351
8.3.6;20.6 Current Triana Applications;353
8.3.7;20.7 Example 1: Distributing GAP Services;354
8.3.8;20.8 Example 2: The Visual GAT;356
8.3.9;20.9 Conclusion;360
8.3.10;20.10 Acknowledgments;360
8.4;21 Java CoG Kit Workflow;361
8.4.1;21.1 Introduction;361
8.4.2;21.2 The Java CoG Kit Karajan Workflow Framework;366
8.4.3;21.3 Work.ow Support for Experiment Management;376
8.4.4;21.4 Conclusion;376
8.4.5;Acknowledgement;377
8.5;22 Workflow Management in Condor;378
8.5.1;22.1 Introduction;378
8.5.2;22.2 DAGMan Design Principles;379
8.5.3;22.3 DAGMan Details;380
8.5.4;22.4 Implementation Status;389
8.5.5;22.5 Interaction with Condor;390
8.5.6;22.6 Integration with Stork;390
8.5.7;22.7 Future Directions;395
8.5.8;22.8 Conclusions;396
8.6;23 Pegasus: Mapping Large-Scale Work.ows to Distributed Resources;397
8.6.1;23.1 Introduction;397
8.6.2;23.2 Workflow Generation for Pegasus;398
8.6.3;23.3 Pegasus and the Target Workflow Execution Environment;399
8.6.4;23.4 Pegasus and Workflow Refinement;402
8.6.5;23.5 Workflow Execution;406
8.6.6;23.6 Adapting the Workflow Mapping to a Dynamic Execution Environment;406
8.6.7;23.7 Optimizing Workflow Performance with Pegasus;408
8.6.8;23.8 Applications;411
8.6.9;23.9 Related Work;413
8.6.10;23.10 Conclusions;414
8.6.11;Acknowledgements;415
8.7;24 ICENI;416
8.7.1;24.1 Introduction;416
8.7.2;24.2 The Workflow Pipeline;423
8.7.3;24.3 Specification;424
8.7.4;24.4 Realization;426
8.7.5;24.5 Execution Environment;431
8.7.6;24.6 Application Interaction;435
8.7.7;24.7 Conclusion;435
8.8;25 Expressing Workflow in the Cactus Framework;437
8.8.1;25.1 Introduction;437
8.8.2;25.2 Structure;438
8.8.3;25.3 Basic Workflow in Cactus;438
8.8.4;25.4 Extensions;442
8.9;26 Sedna: A BPEL-Based Environment for Visual Scientific Workflow Modeling;449
8.9.1;26.1 Introduction;449
8.9.2;26.2 Modeling Scientific Workflows;451
8.9.3;26.3 Scientific Workflow Editor;457
8.9.4;26.4 Case Study: Polymorph Search;465
8.9.5;26.5 Related Work;468
8.9.6;26.6 Lessons Learned and Future Work;469
8.9.7;26.7 Acknowledgments;470
8.10;27 ASKALON: A Development and Grid Computing Environment for Scienti fic Workflows;471
8.10.1;27.1 Introduction;471
8.10.2;27.2 Work.ow Case Study and Grid Infrastructure;472
8.10.3;27.3 Work.ow Generation;474
8.10.4;27.4 Resource Manager;477
8.10.5;27.5 Scheduler;479
8.10.6;27.6 Execution Engine;484
8.10.7;27.7 Overhead Analysis;486
8.10.8;27.8 Conclusions;491
8.10.9;27.9 Acknowledgments;492
9;Part IV Future Requirements;494
9.1;Looking into the Future of Workflows: The Challenges Ahead;496
9.1.1;1 User Experience;496
9.1.2;2 Workflow Languages and Representations;498
9.1.3;3 Workflow Compilers;499
9.1.4;4 Workflow Enactors or Executors;500
9.1.5;5 Debugging;501
9.1.6;6 Execution Environments;501
9.1.7;7 The Big Question;502
10;References;504
11;Index;536