What data engineering is

Why this matters

Every organisation produces data in more than one place: a billing system, an events log, a catalogue service, a spreadsheet someone maintains by hand. Each was built for its own job, not for analysis, and each exports data in its own format, on its own schedule, with its own quirks. Left alone, that data is almost useless for answering a question that spans systems — "how many minutes of Jazz did users in Germany listen to last Tuesday?" — because nothing agrees on what a user is, what a play is, or what a clean number looks like.

Data engineering is the work of turning those scattered, messy feeds into datasets the rest of the organisation can trust and re-use. It is less about clever analysis and more about *plumbing done well*: getting data in reliably, making it clean and consistent, checking it, storing it in a shape that is easy to query, and — the part beginners underestimate — being able to run the whole thing again tomorrow and get the right answer without creating duplicates or losing yesterday's work.

You will learn this by building one small pipeline for Riverbank Media, a fictional audio-streaming service. You are its first data engineer. Three feeds land in a folder, and none of them is clean.

Concepts

A data pipeline is an ordered set of steps that moves data from its sources (where it is produced) to a sink (where it is used), transforming it on the way. This course uses a layered pipeline, which is the standard shape:

• Landing / raw: the files exactly as they arrived, stored untouched. If anything downstream is

wrong, you can always go back to the raw truth.

• Staging: the raw data loaded into database tables, still as text, still unjudged. Staging is where

bulk loading happens.

• Clean: typed, standardised, de-duplicated data — one row per real thing, with consistent casing,

parsed dates and numbers, and known problems fixed or removed.

• Warehouse (the model): the clean data arranged for analysis, usually as a star schema of facts

and dimensions (Lesson 6).

• Mart / published: focused, aggregated tables built for a specific business question, safe for

analysts and dashboards to read.

Each layer has one job, and data only flows forward. This makes failures easy to locate: if a number is wrong, you can check each layer in turn instead of debugging one enormous query.

Batch vs streaming. A batch pipeline processes data in chunks on a schedule — for example, "load yesterday's play file every morning at 06:00". A streaming pipeline processes each event within seconds of it happening, continuously. Batch is simpler, cheaper, easier to reason about, and correct for the vast majority of analytics; you reach for streaming only when a decision genuinely cannot wait for the next batch (fraud blocking, live operational dashboards). Riverbank's play data is analysed the next day, so this course is batch — the right and boring choice. A good rule: *start with batch; justify streaming.*

Idempotency and re-runnability. Pipelines fail — a file arrives late, a server reboots mid-load, a bug is found and fixed. So a pipeline must be safe to run again. A step is idempotent if running it twice has the same effect as running it once: no duplicated rows, no double-counted revenue. The single most important property you will build in this course is that re-running the whole pipeline on the same data inserts nothing new. We prove it in Lesson 7, and again in the project.

Riverbank's three feeds, previewed:

• users.csv — user accounts, a dimension source.
• tracks.json — the track catalogue, a dimension source (its schema changed mid-year).
• plays_YYYY-MM-DD.csv — one file of play events per day, the fact source.

Guided exercise

You need python3 and PostgreSQL 16 on your lab machine db01. First, confirm the database server is there. You sign in to db01 as analyst, and that is also your database role; your password is already in ~/.pgpass, so psql does not ask for it. The riverbank database does not exist yet — you create it in Lesson 3 — so connect to the server's own postgres database for now.

1. Connect and check the version.

``bash psql -h localhost -U analyst -d postgres -c "SELECT version();"

``text PostgreSQL 16.15 (Ubuntu 16.15-0ubuntu0.24.04.1) on x86_64-pc-linux-gnu, ...

The exact build string names your lab's PostgreSQL package; what matters is that the major version is 16.

1. Generate the raw feeds. The generator is deterministic, so your files match the ones in this course.

``bash cd ~/riverbank-pipeline python3 dataset/generate.py --out landing

``text wrote 633 user rows, 205 track records, 14 play files, 10486 play rows to landing/

1. Look at a play file. Notice the header and the shape of an event.

``bash head -3 landing/plays_2026-08-01.csv

``text play_id,user_id,track_id,played_at,seconds_played,device P0000439,U00403,T0036,2026-08-01 05:01:25,55,android P0000600,U00260,T0167,2026-08-01 02:21:23,332,android

1. Now look at a file from the second week. The schema drifted — played_at became ts, and device is gone:

``bash head -1 landing/plays_2026-08-08.csv

``text play_id,user_id,track_id,ts,seconds_played

1. Find some bad rows by eye. Real feeds are full of them:

``bash grep -nE ",(-[0-9]+|twelve|)," landing/plays_2026-08-01.csv | head -3

``text 55:P0000645,U00227,T0110,2026-08-01 14:07:15,-85,desktop 118:P0000646,U00157,T0118,2026-08-01 04:55:47,-69,desktop 403:P0000644,U00351,T0031,2026-08-01 10:56:46,twelve,android

Negative numbers of seconds, and the word twelve where a number belongs. Empty values are in there too:

``bash grep -nE ",," landing/plays_2026-08-01.csv | head -1

``text 521:P0000643,U00196,T0196,2026-08-01 09:35:56,,desktop

These are the kinds of problems the pipeline must catch — never load blindly.

1. Peek at the catalogue and spot the schema drift:

``bash head -20 landing/tracks.json

One record carries length_seconds, the next carries duration_sec; one duration is even a string, "318". Your pipeline will reconcile both into a single clean number.

Troubleshooting

`psql: could not connect to server` → PostgreSQL is not running or you used the wrong host → start it (sudo systemctl status postgresql@16-main) and connect with -h localhost; on the lab it listens locally.

`FATAL: database "riverbank" does not exist` → you have not built it yet → that is expected before Lesson 3; connect to -d postgres for now, or run python3 pipeline/run_pipeline.py --init.

`python3: command not found` → you are not on db01 or Python is not installed → this course runs on db01, which has python3 3.12; check with python3 --version.

`generate.py` wrote different counts than shown → you passed a different --seed, --days or --start → use the defaults; the course depends on seed 20260913.

`grep` shows no rows → your shell may quote the pattern differently → the point is only to see that raw data contains junk; open the file in an editor if grep misbehaves.

Check your understanding

1. A dashboard must show fraud alerts within two seconds of a transaction. Batch or streaming, and why?
2. Why keep the raw landing files after they have been loaded into the database?

Summary and next step

• Data engineering makes scattered, messy feeds into trustworthy, re-usable datasets.
• A layered pipeline (raw → staging → clean → warehouse → mart) gives each step one job and makes

failures easy to find.

• Default to batch; justify streaming. Design every step to be safe to run again.

Next, in *Sources and ingestion*, you set up a landing area and think clearly about where data comes from and how to pull it in without losing or corrupting it.

References

• PostgreSQL 16 Documentation — https://www.postgresql.org/docs/16/index.html (accessed 2026-09-13)
• Python 3.12, csv — CSV File Reading and Writing — https://docs.python.org/3.12/library/csv.html (accessed 2026-09-13)