Tech Stack
Description
A self-initiated data engineering project focused on the fundamentals of a clean, testable ETL pipeline rather than a large stack.
The pipeline reads a raw CSV, deduplicates by ID (first occurrence wins), normalizes dates and casts types with pandas, then loads clean rows into a data table and rejected duplicates into a data_reject table in DuckDB. Clean records are exported to timestamped JSON and rejects to CSV.
- Built an extract-clean-load pipeline in Python with pandas and DuckDB, separating clean and rejected records into distinct tables.
- Implemented deduplication, date normalization (DD/MM/YYYY → ISO), and list-field parsing with structured logging.
- Containerized the workflow with Docker and docker-compose, mounting source/target as volumes for host output.
- Wrote 17 pytest unit tests covering deduplication logic, date normalization, type casting, and output file format.
Page Info
Data Cleansing Pipeline
Splits clean rows from duplicates, persists both to DuckDB, and writes deduplicated data to JSON and rejected rows to CSV — all reproducible via Docker and verified with a pytest suite.
Live Demo
Try the Pipeline
Edit the messy CSV below, then run the cleansing logic in your browser.
Deduplicates by id (first occurrence wins) and normalizes dates to ISO — the same logic as the Python pipeline, running client-side.