The Job of a Data Pipeline

The core idea

A data pipeline moves data from where it is created to where it is useful, changing its shape along the way so that people and systems can rely on it. That sentence hides three separate jobs: moving data between systems, reshaping it so it answers questions, and guaranteeing that it arrives correctly and on time. Data engineering is the discipline of doing all three repeatably.

Why it matters

Every report, dashboard, forecast, and model in a business sits on top of some pipeline. If the pipeline is late, the report is stale. If the pipeline is wrong, every decision made from the report is wrong, and nobody notices until something expensive happens. Good data engineering is invisible when it works and extremely visible when it fails, which is why the discipline leans so heavily on reliability practices.

The standard stages

1. Source: operational systems such as a web app database, a payment processor, a CRM, or log files.
2. Ingestion: extracting data from the source and landing it, usually unchanged, in a storage layer you control.
3. Storage: a data warehouse, lake, or lakehouse where raw and processed data live.
4. Transformation: cleaning, joining, and modeling the raw data into tables shaped for analysis.
5. Serving: dashboards, reports, exports, or downstream applications that consume the modeled tables.
6. Orchestration and observability: the scheduling, dependency management, monitoring, and alerting that wrap all of the above.

Worked example

An online subscription business wants a daily view of active subscribers by plan. The source is the production database’s subscriptions table plus the payment processor’s events. Ingestion copies both into the warehouse every night. Transformation joins them, resolves cases where a payment failed but the subscription row still says active, and produces a table with one row per subscriber per day and their status. Serving is a dashboard that reads that table. Orchestration runs ingestion at 2 a.m., transformation at 3 a.m., and sends an alert if either step fails or if the subscriber count moves more than 20 percent day over day.

Common mistakes

• Building a dashboard directly on the production database, which slows the application and bakes business logic into chart settings.
• Transforming data during ingestion so the raw record is lost and cannot be reprocessed when the logic changes.
• Skipping orchestration and running scripts by hand ‘for now’.

Action Step: Pick one report your organization relies on. Trace it backward through the six stages and write down what exists at each one. Note every stage where the answer is ‘nothing’ or ‘a person does it manually’.