Data Engineering and Backend Engineering
The backend produces transactions, events and logs as a side effect of serving users. We are its downstream consumer, and it usually does not know that.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does a data platform need from the services that produce its data, and what is unreasonable to ask of them?
Two consumers with opposite needs. The analyst needs the backend's data to be complete, stable and semantically documented. The backend team needs to ship features without a second team's reporting stack constraining every migration. Almost every friction on this boundary is those two needs meeting without a contract (Data Contracts).
The unit that crosses is one durable business fact — an order placed, a payment captured, a subscription cancelled. The backend stores it as rows in one or more tables inside a transaction; we want it as one immutable record with a key, a time and a version. Those are not the same shape, and the translation between them is where most of the work on this boundary lives.
Read the backend's database directly — a replica, or the same instance out of hours — and treat its tables as the data model. It requires nothing of the backend team, it is available today, and for a single service with a stable schema it works genuinely well (Read Replicas From the Application).
A backend refactor splits orders into orders and order_lines. It is a correct migration, it passed every backend test, and every analytical model that read orders.total breaks or, worse, keeps running against a column that no longer means what it did (Schema Migrations from the Application Side, Semantic Changes).
- A backend refactor splits
ordersintoordersandorder_lines. It is a correct migration, it passed every backend test, and every analytical model that readorders.totalbreaks or, worse, keeps running against a column that no longer means what it did (Schema Migrations from the Application Side, Semantic Changes). - The service publishes an event *and* writes to its database, in that order, without a transaction spanning both. A crash between them loses the event silently — the order exists and analytics never hears about it (The Dual Write Problem, The Transactional Outbox).
- An event is named
OrderUpdatedand carries the full current row. Nobody downstream can tell what changed or why, so every consumer re-derives intent by diffing, and each derives it slightly differently (Commands vs Events, Naming Events). - A retry after a gateway timeout produces a second
PaymentCapturedevent with a fresh id. The backend is behaving correctly; the warehouse counts the payment twice (At-Least-Once Delivery, Duplicate Detection). - The backend adds a feature flag and half the traffic starts writing a different
statusvalue. No schema changed, no event changed, and the funnel metric silently splits into two populations (Feature Flags: Rollout, Kill Switches and Debt, Data Quality). - Analytics starts running against the primary because the replica lagged during a launch, and a scan takes capacity from the checkout path (Workload Isolation).
What is actually happening
- A backend produces three kinds of data as a by-product of its actual job, and they are not interchangeable. State lives in its database and is authoritative for what is true now. Events are records of what happened and are authoritative for history. Logs and traces are operational telemetry, are usually sampled, and are authoritative for nothing analytical (The Event Log, Ingestion Sources).
- The transactional outbox is the mechanism that makes events trustworthy. The service writes the business row and the outbox row in one transaction, and a separate process publishes from the outbox. That converts "did the event get sent" from a distributed-systems problem into a local one, and it is the single most valuable thing a backend team can do for its data consumers (The Transactional Outbox).
- Commands and events are different things and confusing them shows up downstream as a modelling problem. A command is a request that may be rejected; an event is a fact that already happened. A stream of commands cannot be summed into a metric because some of them did not take effect (Commands vs Events).
- Backend idempotency keys and pipeline deduplication are the same idea at two layers. The backend uses a key to make a retried request harmless; we use a business key to make a redelivered event harmless. When the backend exposes its idempotency key in the event, our job becomes trivial (Idempotency Keys: The Mechanism, Deduplication).
- Reading the backend's database couples you to its internals. The tables are the service's private implementation, and every argument about whether a migration is allowed is really an argument about whether that coupling was ever agreed (Schema Leakage, Operational vs Analytical Models).
- The backend is also a data consumer, not only a producer. Search indexes, recommendation features, denormalized read models and pricing tables are frequently produced by a pipeline and served by a service, and that direction has its own freshness contract (Keeping a Search Index in Sync, Feature Pipelines).
- Application logs are the worst analytical source and the most commonly reached for. They are unstructured or semi-structured, sampled under load, retained briefly, and shaped by whoever was debugging that week — none of which is a defect in a log (Structured Logging: Fields a Program Can Read, CSV, JSON and Their Limits).
Where the boundary sits
Backend Engineering asks how a service that handles requests is built, secured and operated. This domain asks what can be concluded, later, from what that service left behind. The overlap is narrow and extremely consequential: it is the set of mechanisms by which a fact that happened inside a transaction becomes a record that outlives it.
The healthiest framing to carry into a conversation with a backend team is that we are a consumer of their published interface, and they get to choose what that interface is — but that we need one, and that their database tables are not it. That reframing turns a permission argument into an API design conversation, which backend engineers are extremely good at.
The reverse direction is real and often forgotten: a search index, a feature table, a denormalized read model and a pricing table are data products that a backend serves to users. When those are stale, it is a product incident with a data root cause.
| We teach | Depth lives in | The mechanism that crosses |
|---|---|---|
| Consuming a service's events as a data source | Backend Engineering: Event-Driven Backends, Writing Event Consumers | The event is the published interface. Its name, key, time and version decide whether history is reconstructable downstream (What a CDC Event Contains). |
| Why an event stream can be trusted at all | Backend Engineering: The Transactional Outbox, The Dual Write Problem | A publish inside the transaction cannot be lost by a crash. A publish beside it can, and the loss is silent on both sides. |
| Deduplicating redelivered facts | Backend Engineering: Idempotency Keys: The Mechanism, Duplicate Detection | The backend's idempotency key, carried into the event, is the business key our merge needs. Without it we invent one and get it wrong. |
| Distinguishing an intent from a fact | Backend Engineering: Commands vs Events, Naming Events | A command may be rejected; an event happened. Summing commands produces a metric that counts attempts and calls them outcomes. |
| Isolating analytics from production | Backend Engineering: Read Replicas From the Application, Making an Existing Service Stateless | A replica protects the primary's CPU and hands you its lag. The isolation is real; the completeness cost is the part that is not discussed (Workload Isolation). |
| Surviving a producer's schema change | Backend Engineering: Schema Migrations from the Application Side, Expand and Contract Migrations | Expand-and-contract works for data consumers too — but only if the consumer list is known, which is what lineage is for (Impact Analysis). |
| Serving a pipeline's output back to a service | Backend Engineering: Keeping a Search Index in Sync, Caching in Backends | A derived index or feature table is a data product with a freshness SLO whose breach is user-visible (Feature Pipelines). |
| Treating logs as a source | Backend Engineering: What a Backend Should Actually Log, Structured Logging: Fields a Program Can Read | Logs are sampled, unstructured and unowned. They are excellent for debugging a pipeline and a poor foundation for a metric. |
The one mechanism worth asking for
If a data team gets to make exactly one request of a backend team, this is it. The pattern is small, well understood, has a name backend engineers recognise, and it closes the only failure on this boundary that produces permanent, unrecoverable, invisible loss.
The failure it closes: a service that commits a row and then publishes an event is doing two writes to two systems with no atomicity between them. A crash in the gap loses the event. Nothing downstream can detect this, because the missing event leaves no trace anywhere — the pipeline is green, the counts look plausible, and the only evidence is a reconciliation against the backend's own state (The Dual Write Problem).
The comparison below is worth having ready verbatim, because the usual counter-argument is "we already publish reliably, we retry". Retries help with a broker that is briefly unavailable. They do nothing about a process that died between the commit and the publish, which is the case that matters.
The handler commits the order transaction, then publishes `OrderPlaced` to the broker, retrying on failure. If the process dies between the two, the order exists and the event never will. If the publish succeeds and the transaction is rolled back, the event describes an order that does not exist.
The handler writes the order row and an outbox row in the same transaction. A separate publisher reads the outbox, publishes, and marks the row sent — retrying freely, because a duplicate publish is harmless to a keyed consumer while a lost one is not.
The two writes in the first version are to different systems with no shared atomicity, so there is a window in which exactly one of them has happened and no participant can tell which. The outbox moves both writes into one transaction the database already guarantees, converting an unsolvable distributed problem into a local one and leaving only duplicate delivery — which a keyed merge absorbs (Upserts and Merges).
Following one field from a handler to a board pack
Backend engineers regularly ask a fair question: "who could possibly be affected by renaming this column?" The honest answer in most organisations is that nobody knows, and that is the actual problem on this boundary — not the rename.
The chain below is what an impact assessment needs to be able to produce on demand. Each node says what it holds and what it can corrupt, and the corruption column is deliberately different at each hop: the backend can change meaning, the capture layer can drop, the transformation can fan out, the model can mis-grain, and the dashboard can re-filter.
Build this graph and the conversation changes shape. Instead of asking a backend team to freeze their schema, you hand them a list of six dashboards and two models, and they schedule the change with a contract window (Data Lineage, Impact Analysis).
- Request handler
holds The business decision and the intent behind the write — the only place where "why" exists.
could corrupt Changes the meaning of a value without changing its type or name; introduces a new enum value behind a flag; reuses an existing value for a new state.
↑ reads from - `orders` table row
holds Current state, authoritative for what is true now.
could corrupt Overwrites the previous state, destroying history that no downstream system captured; a migration renames or splits the column.
↑ reads from - Outbox row / change record
holds One immutable record of the transition, with key, time and version.
could corrupt Loses the record entirely if published outside the transaction; emits a full-row snapshot instead of a transition, so intent must be re-derived downstream.
↑ reads from - Event log partition
holds The durable, replayable sequence for that key.
could corrupt Reorders relative to the key's own history if the partition mapping changed; deletes unread history when retention passes (Retention and Replay).
↑ reads from - Raw landing
holds Exactly what arrived, untouched, including fields nobody uses yet.
could corrupt Nothing, if it is genuinely immutable — which is its entire purpose. It corrupts everything downstream permanently if it was "cleaned" on the way in (The Raw Landing Zone).
↑ reads from - Staging model
holds Typed, deduplicated, one row per business fact.
could corrupt A cast that silently nulls; a deduplication keyed on the wrong column; a filter on a status value that has just gained a sibling (Nullability & Defaults).
↑ reads from - `fct_orders`
holds One row per order at a declared grain, with measures and foreign keys.
could corrupt A join to a dimension with duplicate keys, fanning out the fact and multiplying every measure (Grain: What Does One Row Represent?, Fact Tables).
↑ reads from - Metric definition
holds The agreed expression of "revenue" as a filter and an aggregate.
could corrupt Two definitions diverging between teams; a status filter that no longer matches the backend's vocabulary (The Metrics Layer).
↑ reads from - Board pack figure
holds A single number with no visible provenance and total apparent authority.
could corrupt A BI-layer filter or join applied after the model, invisible to every test the data team wrote (Where Did This Number Come From?).
The backend can only reason about the first two nodes. Everything after them is ours to make legible, and lineage is how the first two get told what they are about to break.
How to build it
Most important first.
- Ask the backend for an explicit event stream with a schema, not for access to its tables. The event is a published interface the backend team can commit to; the table is not (Data Contracts, Event-Driven Backends).
- Insist on the outbox pattern, or on log-based capture, and nothing that publishes outside the transaction. Every other correctness measure downstream is compensating for a gap this closes at the source (The Transactional Outbox, Change Data Capture).
- Require a stable business key, an event time distinct from the publish time, and a version or sequence number per entity. Those three fields are what make an event stream reprocessable (Event Keys and Partition Assignment, Event Time).
- Name events after facts in the business's language, one event per meaningful transition, and resist the single
EntityUpdatedevent that pushes interpretation onto every consumer (Naming Events, What a CDC Event Contains). - Where you must read the database, read it through an agreed view rather than the base tables, so the backend can refactor behind it. A view is a cheap contract (Contract Enforcement).
- Give the backend team something back: a lineage view showing which dashboards depend on which of their fields, so an impact assessment is available before the migration rather than after the incident (Impact Analysis, Data Lineage).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The backend guarantees the durability and constraint-consistency of its own state at commit, and nothing about anything it publishes unless the publish is in that transaction (The Transactional Outbox).
- An outbox-published event stream guarantees that every committed business fact is published at least once, in the order the outbox rows were written. It does not guarantee ordering across entities and does not guarantee that consumers ever process it (At-Least-Once Delivery).
- A directly-read database guarantees whatever the isolation level and replica lag allow, and guarantees nothing at all about schema stability, because you are reading something nobody promised to keep stable.
- Logs guarantee nothing analytical. They are best-effort, frequently sampled, and their format is owned by whoever last edited that line of code (What a Backend Should Actually Log).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The boundary check is per-entity completeness against the backend's own count: for a closed period, compare distinct business keys in the warehouse with the backend's count of the same facts, obtained from the backend rather than from the pipeline.
- It misses everything where the backend and the pipeline share an upstream error, and it misses semantic drift entirely — a
statusvalue that changed meaning reconciles perfectly on counts. - The complementary check is asserting the *set* of enum values in a column against the contract, which catches a backend quietly introducing a new state and does not catch an existing state being reused for a new purpose (Data Tests).
- Event-driven capture makes freshness a function of consumer lag rather than of a schedule, which is the main reason to prefer it even when nobody needs minute-level data today.
- Database polling is bounded below by its interval and, more importantly, cannot see states that existed and were overwritten between two polls — a completeness limitation, not a latency one (CDC vs Polling).
- When the pipeline serves the backend rather than the other way round, freshness becomes a user-facing property: a stale search index or feature table is a product bug, not a reporting inconvenience (Keeping a Search Index in Sync).
- Backend schema changes are frequent and correct. The question is never how to stop them, it is how to find out before the dashboard does — which is contract enforcement plus impact analysis, not a change-approval process (Contract Enforcement, Impact Analysis).
- Expand-and-contract migrations are the backend pattern that makes this survivable: add the new field, dual-write, migrate consumers, then remove. It works for data consumers too, provided somebody told them the window had started (Expand and Contract Migrations, Backward Compatibility).
- The changes that hurt are the ones with no schema signal: a reused enum value, a column that changes unit, a flag that changes which code path writes the row. Only a documented contract and a distribution test have any chance (Semantic Changes, Distribution Tests).
- If events are published from an outbox and retained, recovery is a replay: reset the consumer position, rebuild the affected models, publish atomically (Replay from the Log, Atomic Publish).
- If the source of truth is the backend's database and history was overwritten, recovery is impossible for the overwritten states. This is the argument for capturing changes rather than states, made concretely (Keeping Raw History: The Recovery Position and the Liability).
- Recovery from a schema-drift incident means re-landing raw for the affected window with the corrected mapping, then re-running the models — which only works if raw was landed before the mapping was applied (The Raw Landing Zone).
What can go wrong
- A dual write losing events on crash, producing an under-count that never resolves and is invisible in every pipeline metric.
- A backend migration silently changing the meaning of a column that a hundred models depend on.
- An analytical query running against a production instance and consuming capacity the product needed.
- Retried requests producing duplicate events that the pipeline counts as separate facts.
- The mitigation failing: a contract test that validates the schema of an event but not the population of its enum, so a new
statusvalue passes and re-splits every funnel (Data Contracts). - A pipeline-produced dataset that the backend serves going stale, turning a data incident into a user-visible product incident (Feature Pipelines).
- "The backend already has the data, so we just need access." Access is not the problem. Meaning, stability, history and completeness are the problems, and none of them is granted by a credential.
- "Events are just database rows in a queue." An event is a fact with a name, a time and an intent; a row is current state. Publishing rows and calling them events is how consumers end up re-deriving intent by diffing (Event vs Snapshot Modeling).
- "We can reconstruct history from the current tables." Only for whatever the tables happen to retain. Anything overwritten is gone, and discovering that is usually a quarter's worth of a question that cannot be answered (Slowly Changing Dimensions).
- "Application logs are a data source." They are an operational artefact — sampled, unstructured and owned by nobody in particular. Building a revenue metric on a log line is a decision to have that metric change when someone improves a debug message.
- "Schema changes are safe as long as the column still exists." A column that keeps its name and type and changes its meaning breaks more dashboards than a dropped column, because a dropped column fails loudly (Semantic Changes).
Operating it
- Outbox depth and publish lag on the backend side, exposed to the data team. It is the earliest signal that events are about to be late or missing (Pipeline Metrics).
- Deploy markers from the backend overlaid on data-quality signals — a data anomaly that starts within minutes of a service deploy has a much shorter list of suspects ("What Changed?" — Deploy Markers and the Invisible Deploys, Deploys Are the First Suspect).
- Distinct-key counts per entity per period, compared to the backend's own counters rather than to the pipeline's previous run (Reconciliation).
- A lineage edge from each backend field to the models and dashboards that consume it, queryable by the backend team before they migrate (Column-Level Lineage).
- At 10x request volume the backend's tolerance for analytical reads on its database goes to zero, and any pipeline still doing that will be told to stop during an incident rather than during a planning cycle.
- At 100x, or at ten services instead of one, the per-service integration approach collapses and the platform needs a standard: one publication mechanism, one schema registry, one naming convention (Schema Registry, Data Platform Engineering).
- Service count scales the *contract* problem, not the volume problem. Twenty services with twenty ad-hoc integrations is an operational burden that grows quadratically with team turnover (Data Ownership).
- Reading a backend database costs the backend: query capacity, connection slots, replica headroom. It is the one cost on this boundary that lands on somebody else's budget and therefore gets noticed late (Workload Isolation).
- An outbox costs the backend a write per business fact and a publisher process. That is a genuine, ongoing cost to another team, and pretending it is free is how the request gets refused.
- On our side the drivers are the usual ones — bytes landed and retained, and how much history a model re-reads per run (What Actually Drives Data Platform Cost).
- Asking for an event stream buys stability, history and decoupling, and costs the backend team implementation and maintenance work for a consumer they do not serve. That trade only closes if the data team makes the value visible.
- Reading the database is available immediately and costs you every future migration. It is the right choice for a short-lived question and the wrong one for a platform.
- Contracts make breakage loud, which means more alerts, more coordination and more work refused at the boundary. The alternative is that breakage is quiet, which is cheaper right up until the quarter's numbers are restated (Contract Enforcement).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat the producer owns meaning, that a publish outside the transaction can be lost, and that a table is not an interface hold for any backend in any language. What varies is which mechanism is available to fix it.
- ORG-SPECIFICWhether the backend team will implement an outbox depends on ownership and incentives, not on architecture. In a company where the data platform reports into product engineering it is a normal ticket; where it does not, log-based capture is often the only mechanism that needs no other team's roadmap.
- SOURCE-SPECIFICA backend you own can be asked for an outbox; a SaaS system you integrate with offers whatever API and webhook semantics it offers, usually with no ordering guarantee and a retry policy you cannot see. The design advice here applies fully to the first and only partially to the second (Ingestion Sources).
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a publish beside a transaction cannot be made safe by retrying — the ambiguity of a failed write to a second system is not an engineering gap, it is a property of the setting.