Pandey / Schoof Building ETL Pipelines with Python

Create and deploy enterprise-ready ETL pipelines by employing modern methods
1. Auflage 2024
ISBN: 978-1-80461-553-9
Verlag: De Gruyter
Format: EPUB
Kopierschutz: 0 - No protection

E-Book, Englisch, 246 Seiten

Develop production-ready ETL pipelines by leveraging Python libraries and deploying them for suitable use casesKey Features - Understand how to set up a Python virtual environment with PyCharm
- Learn functional and object-oriented approaches to create ETL pipelines
- Create robust CI/CD processes for ETL pipelines
- Purchase of the print or Kindle book includes a free PDF eBook
Book DescriptionModern extract, transform, and load (ETL) pipelines for data engineering have favored the Python language for its broad range of uses and a large assortment of tools, applications, and open source components. With its simplicity and extensive library support, Python has emerged as the undisputed choice for data processing. In this book, you’ll walk through the end-to-end process of ETL data pipeline development, starting with an introduction to the fundamentals of data pipelines and establishing a Python development environment to create pipelines. Once you've explored the ETL pipeline design principles and ET development process, you'll be equipped to design custom ETL pipelines. Next, you'll get to grips with the steps in the ETL process, which involves extracting valuable data; performing transformations, through cleaning, manipulation, and ensuring data integrity; and ultimately loading the processed data into storage systems. You’ll also review several ETL modules in Python, comparing their pros and cons when building data pipelines and leveraging cloud tools, such as AWS, to create scalable data pipelines. Lastly, you’ll learn about the concept of test-driven development for ETL pipelines to ensure safe deployments. By the end of this book, you’ll have worked on several hands-on examples to create high-performance ETL pipelines to develop robust, scalable, and resilient environments using Python.What you will learn - Explore the available libraries and tools to create ETL pipelines using Python
- Write clean and resilient ETL code in Python that can be extended and easily scaled
- Understand the best practices and design principles for creating ETL pipelines
- Orchestrate the ETL process and scale the ETL pipeline effectively
- Discover tools and services available in AWS for ETL pipelines
- Understand different testing strategies and implement them with the ETL process
Who this book is forIf you are a data engineer or software professional looking to create enterprise-level ETL pipelines using Python, this book is for you. Fundamental knowledge of Python is a prerequisite.

Autoren/Hrsg.

Pandey, Brij Kishore

Schoof, Emily Ro

Fachgebiete

• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Automatische Datenerfassung, Datenanalyse
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Data Mining

Inhaltsverzeichnis

Table of Contents - A Primer on Python and the Development Environment
- Understanding the ETL Process and Data Pipelines
- Design Principles for Creating Scalable and Resilient Pipelines
- Sourcing Insightful Data and Data Extraction Strategies
- Data Cleansing and Transformation
- Loading Transformed Data
- Tutorial – Building an End-to End ETL Pipeline in Python
- Powerful ETL Libraries and Tools in Python
- A Primer on AWS tools for ETL Processes
- Tutorial – Creating an ETL Pipeline in AWS
- Building Robust Deployment Pipelines in AWS
- Orchestration and Scaling in ETL Pipelines
- Testing Strategies for ETL pipelines
- Best Practices for ETL Pipelines
- Use Cases and Further Reading

Über Autor(innen)

Pandey Brij Kishore :

Brij Kishore Pandey stands as a testament to dedication, innovation, and mastery in the vast domains of software engineering, data engineering, machine learning, and architectural design. His illustrious career, spanning over 14 years, has seen him wear multiple hats, transitioning seamlessly between roles and consistently pushing the boundaries of technological advancement. He has a degree in electrical and electronics engineering. His work history includes the likes of JP Morgan Chase, American Express, 3M Company, Alaska Airlines, and Cigna Healthcare. He is currently working as a principal software engineer at Automatic Data Processing Inc. (ADP). Originally from India, he resides in Parsippany, New Jersey, with his wife and daughter. Schoof Emily Ro :

Emily Ro Schoof is a dedicated data specialist with a global perspective, showcasing her expertise as a data scientist and data engineer on both national and international platforms. Drawing from a background rooted in healthcare and experimental design, she brings a unique perspective of expertise to her data analytic roles. Emily's multifaceted career ranges from working with UNICEF to design automated forecasting algorithms to identify conflict anomalies using near real-time media monitoring to serving as a subject matter expert for General Assembly's Data Engineering course content and design. Her mission is to empower individuals to leverage data for positive impact. Emily holds the strong belief that providing easy access to resources that merge theory and real-world applications is the essential first step in this process.