Chadha Data Engineering with Databricks Cookbook

Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake
1. Auflage 2024
ISBN: 978-1-83763-206-0
Verlag: De Gruyter
Format: EPUB
Kopierschutz: 0 - No protection

E-Book, Englisch, 438 Seiten

Written by a Senior Solutions Architect at Databricks, Data Engineering with Databricks Cookbook will show you how to effectively use Apache Spark, Delta Lake, and Databricks for data engineering, starting with comprehensive introduction to data ingestion and loading with Apache Spark.
What makes this book unique is its recipe-based approach, which will help you put your knowledge to use straight away and tackle common problems. You'll be introduced to various data manipulation and data transformation solutions that can be applied to data, find out how to manage and optimize Delta tables, and get to grips with ingesting and processing streaming data. The book will also show you how to improve the performance problems of Apache Spark apps and Delta Lake. Advanced recipes later in the book will teach you how to use Databricks to implement DataOps and DevOps practices, as well as how to orchestrate and schedule data pipelines using Databricks Workflows. You'll also go through the full process of setup and configuration of the Unity Catalog for data governance.
By the end of this book, you'll be well-versed in building reliable and scalable data pipelines using modern data engineering technologies.

Autoren/Hrsg.

Chadha, Pulkit

Fachgebiete

• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Datenbankdesign & Datenbanktheorie
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Automatische Datenerfassung, Datenanalyse
• Mathematik | Informatik EDV | Informatik Informatik Mensch-Maschine-Interaktion Informationsarchitektur
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Data Warehouse

Inhaltsverzeichnis

Table of Contents - Data Ingestion and Data Extraction with Apache Spark
- Data Transformation and Data Manipulation with Apache Spark
- Data Management with Delta Lake
- Ingesting Streaming Data
- Processing Streaming Data
- Performance Tuning with Apache Spark
- Performance Tuning in Delta Lake
- Orchestration and Scheduling Data Pipeline with Databricks Workflows
- Building Data Pipelines with Delta Live Tables
- Data Governance with Unity Catalog
- Implementing DataOps and DevOps on Databricks

Über Autor(innen)

Chadha Pulkit :

Pulkit Chadha is a Sr. Solutions Architect at Databricks. He has over 12 years of experience working in Data Engineering. With his expertise in building and optimizing data pipelines using various tools and technologies Pulkit has worked with enterprises in various industries, including healthcare, Media and Entertainment, Hi-Tech, and Manufacturing providing data engineering solutions to meet enterprises' unique business needs. His work history includes the likes of Dell Services, Adobe, and Databricks. Pulkit holds a Masters's Degree in Management Information Systems from Eller College of Management at the University Of Arizona and has several cloud certifications in data analytics.