Data Engineering
How do I design reliable data systems that collect, move, transform, validate, store, model, serve and govern data at scale? Not which tool to use — what kind of data is flowing, how fresh it must be, what guarantees are required, how schema changes propagate, how failures are recovered, and how quality is measured.
Dashboard Revenue = 1,245,892
Where did this number come from? Which model produced it, and what does one row of that model represent? Which source system is authoritative? How fresh are the inputs? Could records be missing, or counted twice? Which schema version produced it? Can I trace the lineage, and can I reproduce the result?
A learner who finishes this domain does not see a number. They see the eight systems it passed through, the four times its grain changed, and the one check that would have caught it if it were wrong.
The order is the argument, and it runs in both directions. Forwards it is a design sequence: what the consumer needs decides the grain, the grain decides the model, the model decides what can be validated. Backwards it is the only productive way to debug — nobody has ever found a data incident by starting at the source.
What can go wrong between the source system and the dashboard
Ask an engineer what breaks in a data pipeline and you will usually hear “the job fails”. Job failures are the easy case: they are loud, attributable, and the orchestrator tells you. These are the expensive ones.
This is the movement-and-trust domain
Data Engineering does not own the depth of its neighbours. It owns what happens to a record between the system that wrote it and the person who believes it.
Backend Engineering builds the service that produced the write, and Database Engineering owns how it is stored, indexed, transacted and replicated. We are their downstream consumer.
Capture, ingestion, raw retention, transformation, validation, modelling, layout, serving, lineage, governance, recovery and cost — and the discipline of knowing what each hop actually promises.
Analytics, finance, product, machine learning and agent systems are consumers. We serve them; we do not become them. Observability & Performance asks why it is slow — a different question, with a different toolkit.
Flagship experiences
The parts of this domain that are not reading.
Break change capture, duplicate an event, delay one, change a schema, crash the transform, create skew, re-run a backfill. Watch what each does to the number on the dashboard — and to which check would have caught it. No two faults fail the same set of checks, which is what makes diagnosis possible at all.
A user clicks "Buy". Follow that fact through the transaction, the write-ahead log, change capture, the event log, raw storage, transformation, a fact table, a warehouse and a mart, until it is a number. The grain changes four times on the way, and each change is a place a correct-looking aggregate becomes wrong.
Revenue is down 40%. Production orders are stable, capture lag is high, warehouse freshness is seven hours, the transform succeeded. Diagnose it before the answer is shown.
Choose a partition key and a query, and see what the reader actually has to touch. Then store the same dataset as CSV, JSON, Avro, Parquet and ORC and see what each one lets you skip.
An event happened at 10:00 and arrived at 10:07. Which window does it belong to, and is it counted? Move the allowed-lateness slider and find out where the trade actually sits.
Orders, customers, products, payments. Choose a fact grain, dimensions, measures and how history is handled — then see which business questions your model answers, which it answers wrongly, and which it cannot answer at all.
A revenue bug has been wrong for six months. Fix the logic, choose the range, recompute and publish — without corrupting the partition that was already correct.
Inject nulls, duplicates, a missing day and a volume drop, then write the checks. Every check has a blind spot, and the exercise is finding out which.
Click a metric and walk upstream one hop at a time, asking the same question at each: is the affected period complete here? The first "no" is where the incident lives.
One key holds most of the rows and one task holds most of the work. Add workers and watch nothing happen. Salt the hot key and watch it move.
Two queries over the same dataset, wildly different amounts of work. Find the driver — and notice that the thing that looks expensive is rarely the thing to fix.
The flagship decision tree. Every answer names why, what it trades away, how it fails, and something simpler you should have ruled out first.
Six sources, six consumers, and a starting point of one nightly CSV. Eleven requirements arrive, then ten things go wrong. Keep the platform trustworthy.
Production situations with the cause unlabelled. Each carries the trap — the wrong fix that looks right and makes the next incident worse.
What each question is really testing, what a strong answer sounds like, and the red flags that separate a remembered rule from a working model.
Learning modules
Twenty-eight modules, from what this discipline actually is once the tools are removed to explaining where a number came from.
What this discipline is once the tools are removed: the journey from an application write to a number on a dashboard, the thirteen things that go wrong along it, and why "the pipeline succeeded" is not evidence that the data is right.
Two workloads with opposite shapes — many small transactions against current state, versus large scans across history. The distinction that explains why analytics moved off the production database in the first place.
Parquet, Avro, ORC and the text formats they replaced. Row groups, column chunks, statistics, encodings — what a format actually stores, and what that lets a reader skip.
Getting data out of systems you often do not control. Batch extracts, incremental windows, streaming producers, and the failure recovery that decides whether a missed hour is recoverable or gone.
Where transformation runs and what that decides. Not a fashion — a question about where compute lives, how much raw history you keep, and what you can reprocess after you find a bug.
Object storage, analytical warehouses, and the table-metadata layer that gave files transactions. Compared on data types, query patterns, governance, cost, openness and tooling — not on marketing.
Where bytes physically sit decides what a query must read. File size, compaction, partitioning, pruning, cardinality, clustering and bucketing — the highest-leverage and least-visible decisions in analytics.
Facts, dimensions, grain and history. The model decides which business questions are easy, which are expensive, and which are answerable but silently wrong.
Cleaning, casting, joining, aggregating and deduplicating — expressed as a dependency graph of tested, documented models rather than a pile of scheduled scripts.
Coordinating work by dependency, state and time. Why a scheduler is not an orchestrator, what a failed task in the middle of a DAG means, and why idempotency is the property that makes re-running safe.
Reading a database's own change log instead of asking it questions. What CDC gives you that polling cannot, what it costs the source, and every way it silently loses or reorders changes.
The durable, partitioned, replayable append-only log as data infrastructure. Topics, partitions, keys, consumer groups, offsets and retention — and why replay is the feature that matters most here.
Continuous computation over unbounded data. Event time versus processing time, windows, watermarks, state, joins, and what "exactly-once" can and cannot mean.
Spark and its relatives from the inside: partitions, stages, tasks and the shuffle. Skew, stragglers and salting — why one task in a thousand decides your job's runtime.
Engines that query data they do not own. Coordinators and workers, pushdown, vectorized execution, and the real limits of federating a query across systems.
How do we know the data is correct enough to trust? Dimensions, tests, distribution checks, freshness and reconciliation — plus what every check still misses.
Producers and consumers agreeing explicitly. Which schema changes are safe, which break silently, and why a change that passes every schema check can still destroy a metric.
Data about data, and the graph that connects it. Discovery, ownership, column-level lineage and impact analysis — the difference between a warehouse and a landfill.
Classification, PII, minimization, retention, access control, masking and deletion — applied to datasets and pipelines rather than to endpoints.
Pipeline health is not data health. Freshness, volume, schema and quality as monitored signals, and the upstream walk that turns "revenue looks wrong" into a cause.
Fixing history without breaking the present. Backfill ranges, late-arriving data, deduplication, merges, replay and the validation that has to happen before you publish.
Retries, checkpoints, atomic publish, partial failure and rollback — plus the SLOs that make freshness a commitment instead of a hope.
Data platforms get expensive quietly. The drivers — bytes scanned, bytes shuffled, bytes retained, hours held, work repeated — and the design decisions that move each one.
Central warehouse, event-driven platform, Lambda, Kappa and mesh, compared by what problem each was a response to and what it costs an organisation to run.
The primitives first, then how BigQuery, Snowflake, ClickHouse, DuckDB and the managed streaming services realise them — architecturally, not from a feature list.
Retrieval corpora, embeddings, evaluation sets and agent traces are data products with schemas, freshness, lineage and cost. Re-embedding is a data migration.
The dashboard is wrong. Working from a number back to its source through models, joins, partitions and ingestion — and the anti-patterns that made it wrong in the first place.
Where this domain touches databases, distributed systems, backends, cloud, delivery, observability and security — and exactly where each of those owns the depth.
Reference
For when you already know roughly what you are looking for.
Ten levels, each stating what you can actually build once you have it.
The searchable index: what a problem sounds like in a meeting, and the mechanism to think of.
Ten pairs people genuinely confuse, with the confusion named.
Where this domain touches databases, backends, cloud, delivery, observability and security — and where each of those owns the depth.