Tech Stack
Description
A self-initiated data engineering project that models a real mining-operations use case — turning raw hauling-trip (ritase) records into analytics-ready marts using the medallion architecture on a modern, containerized data stack.
Apache Airflow (CeleryExecutor, Redis broker, Postgres metadata DB) orchestrates a daily DAG: ingest_bronze → transform_silver → transform_gold → load_postgres. PySpark handles the transformations, reading and writing parquet across Bronze/Silver/Gold zones in MinIO (S3-compatible storage), and the Gold marts are upserted into a PostgreSQL warehouse. The entire stack runs in Docker Compose.
- Designed a Bronze → Silver → Gold medallion pipeline: raw ingest, cleaning/validation/deduplication, then aggregation into shift, daily, equipment, route, and supervisor marts.
- Orchestrated the workflow with Airflow (CeleryExecutor + Redis) and processed data with PySpark via hadoop-aws S3A access to MinIO.
- Loaded curated Gold marts into a PostgreSQL data warehouse with idempotent upserts, and containerized every service with Docker Compose.
- Built with production hygiene in mind: environment-based credentials (no hardcoded secrets), a failure-callback hook, CI for lint + DAG-import validation, and DAG integrity tests.
Page Info
Medallion ETL Pipeline
Ingests raw Excel data to a Bronze layer, cleans and validates it into Silver, aggregates business marts in Gold, and upserts them into a PostgreSQL data warehouse — orchestrated end to end by Airflow.
Live Demo
Try the Pipeline
Run the Bronze → Silver → Gold medallion logic on sample ritase data — in your browser.
Bronze · raw records (8)
| truck | shift | equipment | route | tonnage |
|---|---|---|---|---|
| HD785-01 | day | Excavator A | Pit-1 | 91 |
| HD785-02 | DAY | Excavator A | Pit-1 | 88 |
| HD785-01 | day | Excavator A | Pit-1 | 91 |
| HD785-03 | night | Excavator B | Pit-2 | 95 |
| HD785-04 | Night | Excavator B | Pit-2 | -5 |
| HD785-05 | night | Excavator B | Pit-2 | 90 |
| HD785-06 | Day | Excavator C | Pit-1 | 84 |
| HD785-07 | day | — | Pit-3 | 78 |
Sample data · the same dedupe, validation, and aggregation logic the PySpark pipeline applies across the Bronze, Silver, and Gold layers.