Coal Mining ETL Pipeline

A medallion-architecture (Bronze → Silver → Gold) ETL pipeline for coal mining ritase (hauling trip) data, orchestrated with Apache Airflow and processed with PySpark over MinIO object storage, landing aggregated marts in a PostgreSQL warehouse. Fully Dockerized.

A self-initiated data engineering project that models a real mining-operations use case — turning raw hauling-trip (ritase) records into analytics-ready marts using the medallion architecture on a modern, containerized data stack.

Apache Airflow (CeleryExecutor, Redis broker, Postgres metadata DB) orchestrates a daily DAG: ingest_bronze → transform_silver → transform_gold → load_postgres. PySpark handles the transformations, reading and writing parquet across Bronze/Silver/Gold zones in MinIO (S3-compatible storage), and the Gold marts are upserted into a PostgreSQL warehouse. The entire stack runs in Docker Compose.

What I built

Designed a Bronze → Silver → Gold medallion pipeline: raw ingest, cleaning/validation/deduplication, then aggregation into shift, daily, equipment, route, and supervisor marts.
Orchestrated the workflow with Airflow (CeleryExecutor + Redis) and processed data with PySpark via hadoop-aws S3A access to MinIO.
Loaded curated Gold marts into a PostgreSQL data warehouse with idempotent upserts, and containerized every service with Docker Compose.
Built with production hygiene in mind: environment-based credentials (no hardcoded secrets), a failure-callback hook, CI for lint + DAG-import validation, and DAG integrity tests.