Engineering
What Is ETL, and What Is a Data Pipeline?
Extract, transform, load: the pattern behind moving data from the systems that produce it into a place where it can be analysed.
Definition
ETL stands for extract, transform, load. It describes a process that pulls data from source systems, reshapes it into a consistent structure, and writes it into a destination such as a warehouse. A data pipeline is the scheduled, repeatable implementation of that process.
The three stages
- Extract: pull from APIs, databases, exports or logs, handling pagination and rate limits
- Transform: clean, normalise, deduplicate, join and validate the records
- Load: write into the destination in a way that can be re-run without duplicating rows
ELT and why the order changed
Modern warehouses are fast enough to do transformation themselves, so many teams now load raw data first and transform inside the warehouse. This is ELT. The advantage is that the untransformed source is preserved, so a change in business logic can be replayed over history rather than requiring a fresh extract.
Properties that matter more than the tooling
- Idempotence: running the same job twice produces the same result, not double the rows
- Incrementality: fetch only what changed, which matters when the source is billed per call
- Observability: every run records what it read, wrote and skipped
- Failure isolation: one bad record fails one record, not the whole batch
A pipeline that cannot be re-run safely is not a pipeline. It is a one-off script with a schedule attached.