ELT Pipeline: Airflow + dbt + Postgres
A fully containerized ELT pipeline moving data between Postgres databases with Airflow orchestration and dbt transformations into analytics-ready models.
Move data, transform it, and orchestrate it, reproducibly.
The goal was a clean, fully containerized Extract-Load-Transform pipeline that moves data from a source Postgres database to a destination and builds analytics-ready tables, with every piece runnable via Docker Compose.
Airflow orchestrates, dbt transforms.
A Python script uses `pg_dump` + `psql` to extract and load the full source database into the destination. dbt then builds models that create clean tables (films, actors, film_actors, film_ratings) with business logic like rating categories and actor aggregations.
Airflow runs both steps as a DAG, ensuring the ELT completes before dbt starts, and the dbt step runs in its own container spawned via Airflow’s DockerOperator.