Data Engineering Framework
A reusable PySpark library for standardising how data pipelines are written across a team — covering the full lifecycle from connection and ingestion through to transformation, testing, and write-back.
Overview
Data engineering teams often end up with pipelines that work but don't scale — each one written slightly differently, with inconsistent patterns for joining, testing, and writing data. This framework solves that by providing a single set of well-tested functions that cover every stage of a pipeline, so building a new one becomes assembly work rather than greenfield engineering each time.
Multi-Source Ingestion
Unified connector pattern across JDBC (SQL Server, Oracle), Azure Blob Storage, Snowflake, Salesforce, and REST APIs.
Transformation Library
Informatica-inspired functions — router, joiner, MD5 checksum, SCD flag generation, and lookup — all built in PySpark.
Regression Testing
Schema comparison, row-level hash diffs, and cell-by-cell mismatch reporting to catch data drift before it reaches production.
Write-Back Patterns
Delta Lake merge/upsert with auto-generated conditions, table swap for full reloads, and Databricks Unity Catalog writes.
Roadmap
Jupyter notebook validating all utility functions locally with PySpark — no cloud credentials required.
Notebook tests on Databricks CE covering Delta Lake merge, Unity Catalog writes, and secret scope integration.
Live Snowflake read/write tests followed by a dbt project modelling ingested data through bronze → silver → gold layers.