Data Pipeline Abstraction
Introduction
In ETL world, data engineers/data analysts often talk about data pipeline, data flow, workflow which are very similar. Many people misunderstand those concepts or are confused by those concepts. Today I’d like to talk about their differences from abstraction perspective. In my opinion, we can classify data pipelines into 2 categories:
- Operator Level Abstraction ( e.g. Spark pipeline / Apache beam pipeline, each node in DAG is a model/dataset. Building this kind of data pipeline is just like object oriented programming)
- Task Level Abstraction (e.g. Airflow pipeline, each node in DAG is a task. Building this kind of data pipeline is just like procedural programing)
Operator Level Abstraction
This abstraction is data oriented which means the abstraction is on the dataset. Each node represents a dataset after applying one operator. The operator type could be categorized as classical Source/Transform/Sink pattern which corresponds to read/process/write data. For this kind of pipeline:
- All the nodes logic is running under the same runtime, this kind of pipeline is bounded with one underlying computing engine (e.g. Spark, Flink)
- Data would move between nodes, and the data movement is controlled by the underlying computing engine(e.g. Spark, Flink).
Here’s one example pipeline which reads data from Kafka, then does some filter processing and finally writes to Hdfs.
Task Level Abstraction
This abstraction is task oriented. Each task could be heterogeneous and running under different runtime (e.g. SparkSql, Shell, Python).
- This kind of pipeline is bounded with a workflow engine (e.g. Airflow)
- Data movement between nodes is not mandatory and is under user’s control, the pipeline runtime framework (Airflow) doesn’t control that, pipeline runtime framework only controls the execution order.
Here’s one example pipeline which first runs one Spark sql job, then runs a shell script, and finally runs one Python script.
Here’s a summary table to compare those 2 kinds of pipelines:
DBT & SqlMesh
At the first glance, you may think dbt & sqlmesh is task level abstraction. But actually they are operator level abstraction. Each node of dbt/sqlmesh represent a dataset. It is not a arbitrary sql script in each node of db/sqlmesh, it is a speical sql script with select statement as the last statement. This select statement represents a dataset. So they are data oriented rather than task oriented.
Mixture Those 2 Kinds of Pipelines
Users can not mix those 2 kinds of pipelines arbitrarily. E.g. The following pipeline is invalid.
But we can still mix those 2 kinds of pipelines by following the specific rule (Put operator level pipeline as a sub node of task level pipeline).e g.
Summary
In conclusion, understanding the distinctions between Operator Level and Task Level abstractions is crucial for designing effective data pipelines in ETL processes. Operator Level pipelines, with their data-centric approach and reliance on a unified computing engine like Spark or Flink, offer streamlined data movement and processing for cohesive workflows. In contrast, Task Level pipelines, orchestrated by workflow engines like Airflow, provide flexibility by supporting heterogeneous tasks and user-controlled data movement, catering to diverse runtime environments. By recognizing these abstraction models — akin to object-oriented versus procedural programming — data engineers and analysts can better select and implement pipelines that align with their specific project requirements and computational needs.
