Data Engineering and DevOps

• A data deployment has two artefacts, not one. The definition — SQL, Python, a schema, a DAG — behaves exactly like software and can be versioned, reviewed, tested and reverted. The materialisation — the rows already written — is state, and reverting the definition does not touch it (Rolling Back Data).
• That is why a data rollback is a backfill. Restoring correctness means recomputing the affected range with the restored logic, validating it, and publishing it atomically. It is a data operation with a cost proportional to history, not a deploy (Planning a Backfill, Validating a Backfill Before You Publish).
• CI for data has to test two different things. The code — does the SQL compile, does the DAG resolve, is the model's contract unchanged — is testable in isolation and fast. The data — do the assertions hold against realistic input — is only testable against data, and a fixture small enough to be fast is usually too clean to be informative (Data Tests).
• Schema versioning is a producer-consumer problem, not a repository problem. A migration is safe or unsafe depending on who reads the column, and the repository does not contain the readers. This is why contracts and lineage do the work that a version number cannot (Data Contracts, Column-Level Lineage).
• Infrastructure-as-code covers the platform, not the data. Recreating buckets, roles, warehouses and schedules from code is genuinely achievable and genuinely valuable; recreating the contents from code is only possible if raw is retained and transformations are deterministic (Infrastructure as Code, Keeping Raw History: The Recovery Position and the Liability).
• Blue-green has a data analogue and it is the strongest deployment pattern in this domain. Write the new version to a separate location, validate it there, and swap the pointer that consumers read. That is what atomic publish is, and it makes the rollback a second pointer swap rather than a recompute (Atomic Publish).
• Idempotency is the property that makes any of this safe to retry. A deploy that half-ran can be re-run only if re-running produces the same result; without that, every failed deployment becomes a manual reconciliation (Idempotent Data Pipelines).

transform/                      <- versioned; `git revert` restores all of this
  models/
    staging/stg_orders.sql        definition of a transformation
    marts/fct_orders.sql          definition, plus its materialisation strategy
  schemas/
    fct_orders.yml                declared columns, types, tests, owner
  metrics/
    revenue.yml                   the agreed expression of a business metric
  macros/ tests/ seeds/
  dbt_project.yml / dag.py        wiring, schedules, dependencies
.github/workflows/ci.yml          what runs on a pull request
infra/                            buckets, roles, warehouse, schedules (IaC)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  everything below is STATE. `git revert` does not touch any of it.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

warehouse/
  analytics.fct_orders            90 days of rows written by the OLD definition
  analytics.dim_customer          SCD2 history that cannot be recomputed if
                                  the source overwrote the states it records
lake/raw/orders/dt=.../           the only input a recompute can start from
broker/orders.v1                  replayable ONLY within its retention window
bi/                               dashboards, notebooks, extracts, reverse-ETL
                                  syncs — consumers the repository cannot see

• Publish every model to a new location and swap a pointer or a view, rather than writing in place. It converts both deployment and rollback into a metadata operation and removes the window where consumers read a partial state (Atomic Publish).
• Make the rollback plan part of the change, not part of the incident. Every pull request that changes a model's output should say which range needs recomputing if it turns out to be wrong (Planning a Backfill).
• Run the change against real production data in an isolated output location before promoting it. Comparing old output with new output over the same input is the single most informative data test there is (Validating a Backfill Before You Publish).
• Enforce contracts in CI: the model's output schema, its declared grain, and the columns downstream consumers depend on. That is the only part of the blast radius the repository can actually see (Contract Enforcement).
• Version schemas and metric definitions as deliberately as code, with the version recorded in the produced data so a row can say which definition made it (The Metrics Layer, Schema Evolution).
• Treat the platform — buckets, roles, warehouses, schedules — as code so an environment is reproducible, and be explicit that this does not make the data reproducible (Infrastructure as Code, State: The File That Makes It Work and the File That Will Hurt You).

• A merged pull request guarantees that the definition in the repository is the intended one, from that moment forward. It guarantees nothing about any data already produced by the previous definition.
• A green CI run guarantees that the assertions you wrote hold against the input you tested against. On a synthetic fixture that is a statement about the fixture (Data Quality).
• An atomic publish guarantees that readers see either the old complete version or the new complete version, never a mixture — and only for the tables it covers, which is why cross-table consistency needs its own mechanism (Rolling Back Data).
• Infrastructure-as-code guarantees that the platform can be recreated. It explicitly does not guarantee that the datasets can be, and conflating the two is how disaster-recovery plans end up untested (Disaster Recovery).