Data Engineering Fundamentals
What this discipline is once the tools are removed: the journey from an application write to a number on a dashboard, the thirteen things that go wrong along it, and why "the pipeline succeeded" is not evidence that the data is right.
Not the tools. The discipline of moving data between systems so that what arrives is complete, correct, explainable and affordable.
Application to PostgreSQL to extract to lake to transformation to warehouse to mart to dashboard — and what each arrow is actually promising.
Thirteen failure classes, each with its own mechanism, its own detector, and a long list of checks that will never find it.
Databases own storage, distributed systems own guarantees, backends own the transactional service, analytics and ML are consumers. We own movement, transformation, modelling, validation and serving.
Source, ingestion, raw, transformation, validation, storage model, serving, consumer, feedback — nine stages that are a design order forwards and a diagnostic order backwards.
One authoritative system per business concept, everything else explicitly a copy — and the discipline that follows once you have said which is which.
Analysts, finance, product analytics, ML, agents and operational read-back each need a different freshness, tolerate a different amount of revision, and fail in a different way. Design from them inward.
Trust is not a feeling about a dashboard — it is a set of questions a consumer can answer without asking you. It is built slowly, lost in one incident, and rebuilt at a much higher price.