Data Engineering Foundations

Data engineering is the work of moving data from wherever it is produced into a shape the rest of an organisation can trust and re-use. This course teaches the core of that craft, hands-on, by building one small pipeline end to end on **PostgreSQL 16** and **Python 3.12** — the same open-source tools Ultiblob runs in production, and nothing heavier. You work as the first data engineer at *Riverbank Media*, a fictional audio-streaming service. Three feeds land in a folder every day: a user export (CSV), a track catalogue (JSON whose schema drifted mid-year), and one file of play events per day. The data is deliberately messy — missing keys, inconsistent casing, duplicate accounts, non-numeric and out-of-range values, unknown references, duplicated events and a renamed column halfway through the month. All of it is synthetic and generated by a script included with the course; no real data is ever used. Across ten lessons you build the pipeline layer by layer: land raw files, load them into staging tables with `COPY`, clean and standardise in SQL and Python, validate every row and **quarantine** the ones that fail, model a small **star schema** with surrogate keys, and publish a business-facing **mart**. You then make the whole thing repeatable: idempotent loads with upserts and a natural-key conflict rule, an ingestion **watermark** for incremental runs and backfills, a small Python runner with logging and retries, and automated data-quality checks that can fail a run. The final project is a working pipeline you can prove is correct: it quarantines every bad row, loads a clean star schema and mart, and — the point of the whole course — inserts **nothing new** when you run it a second time. Every command, query and number in the course was executed against PostgreSQL 16.15; the output you read is the output that was observed.