O'Neal | Data Engineering Fundamentals for ML Pipelines | E-Book | www.sack.de
E-Book

E-Book, Englisch, Band 3, 429 Seiten

Reihe: ML Engineer's Journey · Series B

O'Neal Data Engineering Fundamentals for ML Pipelines

An Engineer's Field Guide to Building ML that Ships, Scales, and Survives Production
1. Auflage 2026
ISBN: 979-8-90498-283-6
Verlag: PublishDrive
Format: EPUB
Kopierschutz: 0 - No protection

An Engineer's Field Guide to Building ML that Ships, Scales, and Survives Production

E-Book, Englisch, Band 3, 429 Seiten

Reihe: ML Engineer's Journey · Series B

ISBN: 979-8-90498-283-6
Verlag: PublishDrive
Format: EPUB
Kopierschutz: 0 - No protection



Most machine learning failures in production are really data failures, and few courses teach you how to prevent them. This field guide gives you the data engineering habits that make ML pipelines fresh, correct, and trustworthy.
Data engineering is the part of machine learning that most courses skip. You learn to train a model, evaluate it, maybe deploy it from a notebook. Then you get to work and someone hands you a feature table that stopped updating three days ago, a training set with a subtle leak, or a validation step nobody wrote. The model is fine. The data is not.
Data Engineering Fundamentals for ML Pipelines is a practical path from spreadsheets to production ML data. It covers the engineering that surrounds a model and decides whether the data arriving at it is fresh, correct, and consistent with what it was trained on. The discipline has a shape: define the data you need before writing code, land raw data before transforming it, gate every transformation with validation, make writes idempotent, monitor what matters, and write the runbook before going on call.
Eleven chapters build that discipline step by step: from SQL queries to pipelines, the six organs of an ML data pipeline, data sources, the SQL and Python tools you will actually use, schemas and data contracts, data quality and validation, orchestration, features and feature stores, storage choices, and monitoring with an on-call mindset. The book closes with a worked example of your first production pipeline.
Try This exercises, Quick Checks, Key Takeaways, starter schemas, and a SQL pattern catalog make it both a course and a reference. Book 3 of the ML Engineer's Journey, it is written for career switchers, early-career data engineers, and ML practitioners who have realized their production problems are really data problems.

O'Neal Data Engineering Fundamentals for ML Pipelines jetzt bestellen!

Autoren/Hrsg.


Weitere Infos & Material


Chapter 1: From SQL Queries to Data Pipelines


Chapter Summary


If you have written SQL, you already know how to ask a database a question and get an answer. A data pipeline is what happens when that question has to be answered again tomorrow, and the day after, automatically, even when you are asleep, on vacation, or have left the company. This chapter makes the single most important mental-model shift in the whole book: from thinking about a result to thinking about a repeatable, observable process that produces that result on a schedule. We define what a pipeline actually is, how it differs from a query, and why that difference changes almost everything about how you write code.

By the end of the chapter, you will be able to look at any data task and decide whether it should be a one-off query, a scheduled batch job, or a streaming process. You will understand why a pipeline must care about things a query never does: partial failure, reruns, late-arriving data, and who gets paged when it breaks. This is the vocabulary the rest of the book assumes you have. The goal is the mental model, not any particular tool.

Opening Scenario


A data analyst named Priya ran an SQL query every Monday morning to refresh the marketing dashboard. It took her about ten minutes, a cup of coffee, and a quick eyeball of the row count before she hit send. When she joined an ML team six months later, her manager asked her to “just automate it.” Priya pasted the query into a Python script, set it to run at 6 a.m. every Monday, and went home feeling accomplished. Three weeks later, the customer-churn model that depended on her output had been training on stale data the entire time. One Monday, the upstream source table was still loading when her job fired at six on the dot. The query returned roughly half the rows. Nothing caught it: no error message, no alert, no log entry, no angry email from a downstream team. The pipeline reported success. It had just returned garbage.

The reason Priya’s manual process had always worked was invisible to her. Every Monday, she had unconsciously performed a sequence of checks: she glanced at the date on the most recent record, checked whether the row count felt wrong, and peeked at a few values before sending the file on. None of that was written down, because none of it felt like work. It felt like instinct. The automated script could not exercise instinct. It ran when the clock said to run, read whatever rows were there, and called it done.

This chapter is about that gap. It is about the things a human brings to a one-off task that a pipeline must explicitly provide for itself, and about the mental model you need before you can design a pipeline that deserves to run unattended. Priya’s story is not unusual. It is the most common entry point into data engineering, and the engineers who internalize the lesson early are the ones who stop chasing silent failures at 2 a.m.

Why It Matters


Almost everyone entering ML data engineering arrives fluent in queries and innocent of pipelines, and the gap between the two is where new engineers lose the most time and credibility. A query that works perfectly in your hands can fail silently in production for reasons unrelated to the SQL being wrong. The source was slower than usual. A column got renamed. A partner system pushed records late. None of these problems surface as errors in a naive script. They surface as wrong model behavior weeks later, in a business meeting, in front of people who trusted the data.

Internalizing the pipeline mental model early means that every later chapter lands as a solution to a problem you already understand, rather than as a ceremony you cannot motivate. Validation makes sense once you have lived through a silent partial load. Idempotency makes sense once you have double-counted revenue on a retry. Monitoring makes sense only after you are the last person to find out that a job broke.

What This Chapter Will Cover


  • What a Data Pipeline Actually Is: A repeatable, scheduled, observable process that moves or transforms data without a human in the loop, and how it differs from a notebook cell or an ad-hoc query.

  • Query Thinking vs. Pipeline Thinking: The same task – “get last week’s active users” – run by hand vs. run unattended, and every assumption the human made that the pipeline must now make explicitly.

  • The Three Shapes: Batch, Scheduled Batch, and Streaming: When to reach for each shape and a simple rule of thumb for choosing.

  • Why “It Worked When I Ran It” Is a Trap: The catalog of things a human does without noticing, and what happens when those checks disappear.

  • Inputs, Outputs, and the Contract Between Them: Every pipeline step as a function with declared inputs and outputs, and why undeclared contracts are the root cause of most pipeline incidents.

  • Failure Is Normal: Designing for the Unhappy Path: Why a pipeline is mostly code for the unhappy path, and what that means practically.

  • Idempotency in One Sentence: Running the same step twice should produce the same result as running it once – with one concrete example of what happens when it does not.

  • Latency, Freshness, and “How New Is New Enough?”: How freshness requirements drive the choice of pipeline shape, and why you should ask the consumer before you decide.

  • Where the ML Part Comes In: Why silent data quality problems are more dangerous in ML than in reporting, and how they travel invisibly into model behavior.

  • A Map of the Rest of the Book: The six organs of a pipeline (Chapter 2) mapped to the chapters that follow.

What a Data Pipeline Actually Is


Most definitions of a data pipeline are either too abstract or too narrow. Here is a working definition that will carry you through this entire book: a data pipeline is a repeatable, scheduled, observable process that moves or transforms data without a human in the loop.

Every word in that definition is doing work. Repeatable means the process runs more than once and produces consistent results each time; a notebook you run once to generate a chart is not a pipeline. Scheduled means something other than a human decides when it runs: a clock, an event, a file arriving, a dependency completing. Observable means there is a record of what ran, when, what it produced, and whether it succeeded; an unobservable process is effectively undebuggable. And “without a human in the loop” means no one needs to be watching or approving for the process to complete its job.

A SQL query you run interactively has none of these properties. It runs when you choose to. It does not keep a record of its own execution. You are, by definition, in the loop. That is not a criticism of SQL queries. Ad hoc queries are the right tool for exploration and answering one-time questions. The limitation is that they do not scale to “run this every morning regardless of whether anyone is at their desk.” The pipeline mental model is about what changes when that requirement is in effect.

Diagram 1.1, Query vs. Pipeline: What Changes When You Remove the Human

Think about what happens between “a human runs a query” and “a process runs a query automatically.” The human arrives at their laptop, maybe glances at when the source table was last updated, runs the query, eyeballs the result, and moves on. The pipeline arrives at its scheduled time, finds whatever is in the source, runs the query, writes the output, and exits. The human’s unconscious behavior, all the checking and validating and sense-making, was the safety net. When you remove the human, you must build a new safety net out of explicit code. That is the core engineering challenge of data pipelines, and it is what the remaining sections of this chapter are about.

One more important point: a pipeline is not defined by its technology. A bash script on a cron job is a pipeline. A directed acyclic graph in Airflow is a pipeline. A series of dbt models run on a schedule is a pipeline. What makes something a pipeline is the behavior pattern, not the tool. The same mental model applies whether you are on your first day with a simple shell script or your fifth year wiring together Snowflake, dbt, and Airflow.

Query Thinking vs. Pipeline Thinking


To make the difference between query thinking and pipeline thinking concrete, let us walk through a single task twice. The task is: produce a table of last week’s active users from a web events table. An active user is anyone who triggered at least one event between Monday and Sunday of the previous calendar week.

In query mode, you open your SQL client, probably in Snowflake or BigQuery or Postgres, and you write something like: SELECT user_id, COUNT(*) AS events FROM web_events WHERE event_date BETWEEN ‘2026-06-15’ AND ‘2026-06-21’ GROUP BY user_id HAVING events >= 1. You run it, the result looks about right, you export it, done in ten minutes.

Now try to automate that query. The first thing you notice is that the date range is hardcoded. You have to replace it with something that computes “last week” relative to whatever day the job runs. The human knows it is Wednesday. The script knows only what you tell it.

Diagram 1.2, The Same Task in Query Mode vs. Pipeline Mode

Once you fix the date range, a second assumption surfaces. Your Monday-morning job runs at 6 a.m. The web events table is loaded by a separate process that finishes around 5:45 a.m. on most days. Most...



Ihre Fragen, Wünsche oder Anmerkungen
Vorname*
Nachname*
Ihre E-Mail-Adresse*
Kundennr.
Ihre Nachricht*
Lediglich mit * gekennzeichnete Felder sind Pflichtfelder.
Wenn Sie die im Kontaktformular eingegebenen Daten durch Klick auf den nachfolgenden Button übersenden, erklären Sie sich damit einverstanden, dass wir Ihr Angaben für die Beantwortung Ihrer Anfrage verwenden. Selbstverständlich werden Ihre Daten vertraulich behandelt und nicht an Dritte weitergegeben. Sie können der Verwendung Ihrer Daten jederzeit widersprechen. Das Datenhandling bei Sack Fachmedien erklären wir Ihnen in unserer Datenschutzerklärung.