Celebal internship final project

Secure Retail Data Lakehouse

A layered ETL pipeline that protects personal data before analysts see it.

Overview

My final project for the Celebal Excellence Internship 2026. A Python batch pipeline that moves retail customer and transaction data through Raw, Bronze, Silver and Gold layers, validating it and removing or protecting personal information at each step, then loads analytics-ready tables into PostgreSQL and produces data-quality reports.

All data in this project is synthetic, generated with Faker.

How it works

Validation of required columns, relationships, duplicates, numbers, dates and email addresses before anything moves downstream.
Card CVVs are dropped in Bronze. Silver masks personal details, redacts addresses and replaces identifiers with salted SHA-256 tokens.
Gold generalises ages, amounts and postal codes, and builds customer and sales summaries.
Idempotent full-refresh loading into PostgreSQL 16, run with Docker Compose.
pytest covers the generation, validation and privacy rules.

Built with

Python
Pandas
PostgreSQL
SQLAlchemy
Docker
pytest