15  Orchestration and Modern Tooling

Once PySpark transformations are mastered, the next step is automating data pipelines and adopting modern industry tooling for declarative transformations and production scheduling.

15.1 1. Modern ETL: dbt + PySpark

dbt (data build tool) decouples analytical transformations from pipeline orchestration: transformations are defined as SQL models, while dbt manages dependency graphs, automated data testing, and documentation generation.

pip install dbt-spark
dbt init my_etl_project
-- models/staging/stg_trips.sql
-- dbt materializes this model as a view or table in Spark / DW
SELECT
    VendorID                    AS vendor_id,
    tpep_pickup_datetime        AS pickup_at,
    trip_distance,
    fare_amount,
    tip_amount,
    total_amount,
    payment_type
FROM {{ source('raw', 'yellow_taxi') }}
WHERE trip_distance > 0
  AND fare_amount   > 2.5
-- models/marts/fct_daily_trips.sql
SELECT
    DATE(pickup_at)             AS order_date,
    HOUR(pickup_at)             AS pickup_hour,
    COUNT(*)                    AS trip_count,
    AVG(total_amount)           AS avg_ticket,
    SUM(total_amount)           AS total_revenue
FROM {{ ref('stg_trips') }}
GROUP BY 1, 2
# models/schema.yml — Automated data quality assertions
models:
  - name: stg_trips
    columns:
      - name: vendor_id
        tests: [not_null]
      - name: total_amount
        tests:
          - not_null
          - dbt_utils.accepted_range:
              min_value: 0
              max_value: 500
dbt run      # executes analytical models
dbt test     # executes data quality tests
dbt docs generate && dbt docs serve   # generates interactive documentation

15.2 2. Orchestration with Apache Airflow

In production environments, ETL pipelines run automatically on defined schedules. Airflow models workflows as Python Directed Acyclic Graphs (DAGs):

# dags/monthly_taxi_etl.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def extract_step(**ctx):
    month_str = ctx['logical_date'].strftime("%Y-%m")
    print(f"Extracting target month: {month_str}...")
    # extraction logic goes here

def transform_step(**ctx): ...
def load_step(**ctx): ...
def validate_step(**ctx): ...

with DAG(
    dag_id="monthly_nyc_taxi_etl",
    start_date=datetime(2023, 1, 1),
    schedule_interval="0 3 1 * *",  # runs on 1st of every month at 3 AM
    catchup=False,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
) as dag:

    t_extract   = PythonOperator(task_id="extract",   python_callable=extract_step)
    t_transform = PythonOperator(task_id="transform", python_callable=transform_step)
    t_load      = PythonOperator(task_id="load",      python_callable=load_step)
    t_validate  = PythonOperator(task_id="validate",  python_callable=validate_step)

    t_extract >> t_transform >> t_load >> t_validate
Tip

Airflow Alternatives: - Prefect — Python-native orchestration framework requiring minimal boilerplate. - Dagster — Data-asset-centric orchestrator emphasizing observability and asset lineage. - Recommended learning path: Start with Prefect + Local PySpark, then transition to Airflow in enterprise deployments.

15.3 3. Summary and Tooling Ecosystem Map

Click on a tool to view its primary usage scenario.

Pipeline Stage Tool Recommended Usage Scenario
Extract PySpark Large data files (>1 GB), cluster compute, Parquet/JDBC sources
Extract Kafka + Debezium Real-time event streaming and Change Data Capture (CDC)
Transform Spark SQL / DataFrame API Complex multi-table joins, window functions, ML feature engineering
Transform dbt Version-controlled SQL modeling over cloud Data Warehouses
Quality Great Expectations Declarative data validation before loading to production marts
Load Delta Lake / Iceberg ACID transactions, upserts, and time travel in the Lakehouse
Load BigQuery / Snowflake Serverless cloud DWs without infrastructure management overhead
Load DuckDB Embedded analytics in local notebooks or clusterless pipelines
Orchestrator Airflow Enterprise-grade production DAG scheduling with strict SLAs
Orquestrador Prefect Lightweight, Python-native orchestration for agile teams

15.4 4. References and Additional Resources