branch · ★★☆

Data Engineering

The role in one breath: an analyst answers questions, while a data engineer makes the data show up, clean itself and refresh on its own — reliably, with nobody babysitting it. It's about reproducibility, real open-source tooling, and architectures that survive a restart. The turning point from analyst to engineer is exactly when 'works on my laptop' becomes 'works for everyone, on a schedule.'

what you do

In practice you build the conveyor raw → clean → mart: pull data from APIs, databases and files, land it in a warehouse, reshape it with tested and documented SQL models, put the whole thing on a schedule with dependencies, retries and alerts, then package the stack into containers so it spins up anywhere with one command.

stack

Environment: WSL2 + venv/uv + Docker/compose. Python for data: pandas/polars, requests/httpx, tenacity. Loading: dlt (plus Airbyte, Kafka/Redpanda for streaming). Transforms: dbt Core (plus SQLMesh). Storage: DuckDB, Postgres, ClickHouse, BigQuery/Snowflake. Formats & lake: Parquet, Iceberg on MinIO/S3. Orchestration: Prefect, Airflow, Dagster, Kestra. BI: Metabase, Superset, Lightdash, Grafana, a Cube semantic layer, and Streamlit.

who it suits

It clicks for people who've outgrown 'just answer questions off ready-made data' and want to build the pipe itself. For those who like order, automation and engineering, and get a kick when something just runs on its own. A natural next step for an analyst leveling up into an engineer, or a backender drifting toward data.

how it differs

The analyst works with data that's already prepared — questions, dashboards, SQL, stats — and doesn't own the plumbing. The data engineer (that's you) builds and keeps that plumbing: reliable delivery and refresh, so the analyst always has something solid to work on. The ML engineer takes the prepared data and trains models and features on it — your pipelines are the ground the whole ML thing stands on.

0

Script + environment

a couple of evenings

One .py that comes up for anyone from a clean environment. This is the foundation of reproducibility — everything else grows from here.

tool WSL2, venv or uv, git, requirements.txt / pyproject, .env for secrets dataset Any open source you actually care about — a CSV/JSON on a topic you'd enjoy digging into: sports, music, games, crypto, movies.
what you do A script + README + requirements.txt that comes up from scratch with a short series of commands on any machine.
Ready for the next level when:
  • You spin up an isolated environment (venv/uv) and lock dependencies with pinned versions
  • You keep code in the Linux filesystem inside WSL, not in /mnt/c, and understand why it's faster
  • You hide secrets in .env and keep .env, .venv and data out of git via .gitignore
  • You use git with intent: meaningful commits, task branches, a readable README
  • The project runs on someone else's machine without any 'oh, just tweak this by hand' steps
1

From API to storage

1–2 weeks

Learn to pull data yourself and land it in a database instead of a CSV on your desktop.

tool requests/httpx, tenacity for retries, DuckDB or Postgres, DBeaver on hand dataset The API of a service you love — whatever hooks you: game stats, weather, transactions, tracks, repos.
what you do A script: API → a table in a database, repeatable and dupe-free on rerun.
Ready for the next level when:
  • You walk the whole pagination (page / offset / cursor), not just the first page
  • You authenticate by token and keep the key out of code, in the environment
  • You add retries with exponential backoff and respect rate limits — 429/5xx don't kill the run
  • You write results into DuckDB/Postgres and can eyeball the tables in DBeaver
  • A rerun doesn't create duplicates — you upsert/merge on a key
2

Transformations in dbt

2–3 weeks

Turn raw tables into clean marts — as code, with tests, docs and lineage.

tool dbt Core, with Postgres or DuckDB as the engine underneath dataset The same raw data already sitting in your database from the last level — you turn it into meaningful metrics for your topic.
what you do A staging → marts layer wired via ref()/source(), with tests and auto-generated docs plus a dependency graph.
Ready for the next level when:
  • You build staging → marts layers and wire models via ref() and source()
  • You cover key columns with tests (unique, not_null) — and they pass green
  • You build at least one incremental model with a correct is_incremental() filter
  • You generate docs and read lineage from them — where each number came from
  • You deliberately pick a materialization (view / table / incremental) per case
3

Grown-up ingest with dlt

2–3 weeks

Replace hand-rolled scripts with a tool that manages schema and incrementals for you.

tool dlt: resources, write_disposition, cursor-based incrementals, automatic JSON normalization dataset A few sources on one topic you like — a couple of APIs plus a pull from a database or files.
what you do A dlt pipeline: several sources → warehouse with incrementals and an auto-managed schema.
Ready for the next level when:
  • You define sources as dlt resources and load them in a few lines of Python
  • You set write_disposition (append / replace / merge) to fit the task
  • You run cursor-based incrementals without hand-wrangling the schema
  • You flatten nested JSON into plain tables automatically
  • You swap the destination (say, DuckDB ↔ Postgres) with almost no code changes
4

Orchestration and scheduling

3–4 weeks

Wire the steps into one process, put it on a schedule, add retries and alerts.

tool Prefect to start, Airflow next (it's asked most in interviews); Dagster and Kestra to explore dataset The same pipeline on your topic you've already built — but now it runs itself, on a schedule.
what you do dlt → dbt → BI on a schedule with failure alerts and a run history in the UI.
Ready for the next level when:
  • You express the pipeline as a flow/DAG with explicit dependencies between steps
  • You put it on a schedule and can backfill missed dates
  • You attach retries to flaky steps instead of 'I'll just rerun it by hand'
  • You catch failures and get an alert — you don't hear about breakage from users
  • You read logs and run history in the UI and see exactly what broke where
5

Containers and production

finish · production stack

Package the whole stack so it comes up anywhere with a single command.

tool Docker + docker-compose, plus data tests and basic observability dataset Your whole end-to-end case on your chosen topic, fully in containers — e.g. a local modern data stack.
what you do The whole pipeline in docker-compose: database + orchestrator + BI, deployed and reproducible.

Finish: docker compose up — and the whole data stack comes up from scratch.

  • You write a Dockerfile for your pipeline and build a working image
  • You bring up a multi-service setup (database + orchestrator + BI) via compose with depends_on
  • The entire stack starts with one command from a clean machine
  • Secrets and configs live in the environment — nothing is baked into the image
  • You have data tests and basic observability: logs, statuses, alerts
  • The project is polished enough to show an employer without wincing