Data Mesh
An organisational model, not an architecture: domain ownership, data as a product, a self-service platform and federated governance — with the operational cost of each stated honestly.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If the central team cannot hold the meaning of a hundred source systems, what has to change organisationally for the teams that do hold it to publish trustworthy data themselves?
A consumer who currently waits in a queue and then receives a dataset modelled by someone who did not understand the domain. What they need is a dataset built by people who do, with a stated grain, a stated freshness and a name to call when it breaks — which is a change in who does the work, not in what technology does it (Who Actually Consumes This Data).
The unit is a data product owned by a domain team — a dataset with a declared grain, an owner, a contract and an SLO, published for consumption outside the team that produced it. Not a table, and not a team. The distinction is the whole idea: a mesh is a set of products with promises, and a set of tables with owners assigned to them is not one (Data Products).
Adopt the mesh: rename the platform team, tell each domain team that it now owns its data, and put a diagram with hexagons on the wall. This happens constantly and it is worth taking seriously, because the underlying instinct — the people who know the domain should model it — is correct.
Domain teams have no data engineering capacity. They own the datasets on paper and nobody in the team can write a tested, monitored, incrementally-processed pipeline, so the datasets degrade with an owner's name attached (Who Owns Data Quality).
- Domain teams have no data engineering capacity. They own the datasets on paper and nobody in the team can write a tested, monitored, incrementally-processed pipeline, so the datasets degrade with an owner's name attached (Who Owns Data Quality).
- Federated governance has no authority. Each domain interprets
customer_idreasonably and differently, and there is no forum that can force convergence, so cross-domain joins silently produce wrong answers (Surrogate Keys). - The self-service platform does not exist yet, so every domain builds its own ingestion, orchestration and testing. The organisation now operates eight partial platforms rather than one, and the central team it disbanded was the only group that knew how (Data Platform Engineering).
- Cross-domain questions become cross-team negotiations. "Revenue by customer segment" needs three domains to agree on entity resolution and a shared time grain, and there is now a meeting where there used to be a join (The Metrics Layer).
- Nobody owns the shared entities. Customer, product and calendar belong to everyone and therefore to no one, and each domain builds its own version (Dimension Tables).
- The change is announced without moving budget, headcount or on-call, so ownership is nominal and every incident still routes to the platform team by default (Data Ownership).
What is actually happening
- The premise is a specific observation about scale: semantic knowledge does not compress. A central team can operate any amount of data, and it cannot hold the meaning of an unbounded number of source systems. Past that point the constraint is knowledge, not capacity, and hiring into the central team does not move it (The Central Warehouse).
- The response is to move the modelling to where the knowledge already is. The team that built the ordering system knows what
order.statusmeans, which states are reachable, which records are test data and which timestamp is authoritative. That team publishing the dataset is the pattern's entire technical argument. - Four principles hold it together, and each is a commitment rather than a component. Domain ownership: the producing team owns the analytical dataset, not only the operational system. Data as a product: what they publish carries an owner, a contract, quality guarantees and an SLO (Data Products). Self-service platform: the capabilities are provided centrally so that domain teams do not each build a platform (The Self-Service Data Platform). Federated governance: a cross-domain forum defines the small set of things that must be identical everywhere, and the platform enforces them computationally (Data Governance).
- It is explicitly not an architecture. Nothing here specifies a storage layer, a processing engine, a file format or a topology. A mesh can be built entirely inside one warehouse with schemas per domain, or across separate lakehouses, and both are equally faithful to the idea. Any diagram claiming to be "the data mesh architecture" has confused an operating model with a deployment (Data Architecture Patterns).
- The costs are the mirror image of the benefits, and they are organisational too. Distributing modelling distributes the modelling *mistakes*, multiplies the number of people who must understand grain and slowly changing dimensions, and converts coordination that used to happen inside one team into coordination between teams (Agreement Costs Round Trips).
- The hardest part is that federated governance must have teeth without becoming the queue it replaced. A forum that can only recommend produces divergence; a forum that must approve every dataset has reinvented the central team with more meetings (Contract Enforcement).
Four commitments, and what each one actually costs
It is worth saying once, clearly: this is not an architecture. It specifies no storage layer, no engine, no format and no topology, and a diagram of hexagons is not an implementation of it. It is a set of four organisational commitments, and an organisation either makes them — with budget, headcount and on-call — or has adopted a vocabulary.
Each commitment has a benefit that is genuinely hard to obtain any other way, and a cost that is usually discovered a year in. The table below states both. Read the third column first: it is the one that decides whether the model is available to you, because three of the four commitments have a prerequisite that must already be true.
Notice how much of the cost column is about people and authority rather than technology. That is not an artefact of how the table was written; it is what the model is. The technical work of publishing a governed dataset is the same whichever team does it. What changes is who is accountable, who is paged, and who decides what a shared identifier means.
| Commitment | What it means concretely | What must already be true | What it genuinely buys | What it costs to sustain |
|---|---|---|---|---|
| Domain ownership | The team that runs the operational system also builds, tests, monitors and is paged for the analytical dataset derived from it. | That team has data engineering capacity, and its budget and on-call rota were changed to reflect the new work. | The person defining the semantics is the person who knows them. This is the failure a central team cannot avoid at high source count. | Data engineering skill in every domain team, and a permanent increase in the number of people who must understand grain, dimensions and incremental processing. |
| Data as a product | Each published dataset carries an owner, schema, semantics, quality guarantees, documentation, an SLO and an access policy (Data Products). | A definition of what a product must contain, and a platform that refuses to publish one without it. | Consumers get promises rather than tables, and a name to call. Discovery and trust both become possible at scale. | Ongoing product work — documentation, SLO measurement, deprecation notices — that domain teams did not previously do and are rarely resourced for. |
| Self-service platform | Ingestion, storage, compute, orchestration, catalog, testing, security and observability are provided centrally as capabilities (Data Platform Engineering). | A platform team already producing capabilities a domain team can use without filing a ticket (The Self-Service Data Platform). | Domain autonomy without every domain building infrastructure. This is what makes the first commitment affordable. | A large fixed central investment that must be made before the model is available, and that does not scale down. |
| Federated governance | A cross-domain forum defines the small set of concepts that must be identical everywhere; the platform enforces them at publish time (Data Governance). | The forum has authority to decide, and the platform can enforce a decision as a check rather than a memo. | Cross-domain joins that mean something, and consistent classification, retention and access across autonomous teams. | Standing meeting time, real arbitration authority, and constant vigilance that the forum does not become an approval queue. |
Where the boundaries fall, and what crosses them
The practical question a mesh has to answer is what happens at the seams. Inside a domain, a data product is an ordinary pipeline with ordinary failure modes. Between domains, every hop is a contract between two teams with different priorities, different release cadences and different on-call rotas.
Walk the chain below and note where ownership changes. Each of those transitions is a place where a change on one side becomes an incident on the other, and where an incident cannot be resolved by one team alone. This is the coordination the model adds, and it is the honest counterweight to the coordination it removes from the central backlog.
The couldCorrupt column is worth reading as a specification for what federated governance must cover. Almost every entry is a shared concept — an identifier, a time grain, a status vocabulary — and almost none of them is domain-local. That is the argument for keeping the globally-governed set small but non-empty, and for enforcing it computationally rather than by agreement (Contract Enforcement).
- Orders domain: `orders_placed` product
holds One row per placed order, with the order-domain's own status vocabulary and event time.
could corrupt A status value added for an internal workflow that consumers read as a customer-visible state; a change in which timestamp defines "placed" (Semantic Changes).
↑ reads from - Payments domain: `payments_settled` product
holds One row per settlement, keyed by a payment identifier with an order reference.
could corrupt A different definition of the settlement date — authorisation versus capture — so revenue lands in a different day than the orders product expects (Event Time).
↑ reads from - Customer domain: `customer_dimension` product
holds One row per customer per validity period, with segment and region.
could corrupt A slowly changing dimension published as current-state-only, so historical joins attach today's segment to last year's orders (Slowly Changing Dimensions).
↑ reads from - Federated governance: shared entity keys
holds The agreed definition and format of
customer_idandorder_idacross all domains.could corrupt One domain resolving a customer by email while another resolves by account id, so the join silently matches a subset and the metric is quietly low (Surrogate Keys).
↑ reads from - Consumer domain: `revenue_by_segment` product
holds One row per segment-day, joining all three upstream products.
could corrupt A join at mismatched grain — one settlement row per order line against one row per order — fanning out and multiplying revenue (Grain: What Does One Row Represent?).
↑ reads from - Executive dashboard
holds One number, with every boundary above now invisible.
could corrupt A filter applied in the BI layer that none of the four owning teams can see or test (Two Dashboards, Two Numbers).
Four owning teams contribute to one number, and no single team can debug it. This is the coordination cost the model introduces; a central warehouse pays the equivalent cost once, at modelling time, in the form of a queue.
Are you actually a candidate?
The model is right for a specific and identifiable situation, and expensive everywhere else. The diagnostic is not "how much data do we have" — it is "can one team still hold what every source means, and does a platform exist that a domain team could publish through today".
Run the options below as a sequence rather than a menu; the first honest "no" is the answer, and it usually names a project that is a prerequisite rather than an alternative. Organisations that skip to the last option because the first three are uncomfortable are the ones that end up with owners who cannot maintain what they own.
One further honest note: a partial adoption is legitimate and common. Moving the two highest-traffic domains to owning their own products while a central core keeps the finance definitions is a defensible steady state, not a failure to commit. The model is a direction on an axis, not a binary (Data Architecture Patterns).
What is true today, before any renaming?
when Requests are engineering-bound rather than knowledge-bound, and delivered models are trusted.
cost Nothing to do. The model solves a problem you do not have, and its costs are all real (The Central Warehouse).
when A domain team cannot publish a governed, tested, monitored dataset without central engineering work.
cost The platform is the project, and it must be funded before ownership can move. Distributing ownership onto teams with no platform produces several partial platforms and no mesh (The Self-Service Data Platform).
when No one in the producing teams can build and maintain a tested pipeline.
cost Hiring or embedding, and a plan for who is on-call. Without this the model transfers accountability to people who cannot act on it (Who Owns Data Quality).
when When two teams disagree about customer_id, nothing decides.
cost Establishing federated governance with real authority, and expressing its decisions as platform checks rather than documents. Skipping this produces divergence that no technology detects (Data Governance).
when The platform is self-service, domains have capacity, governance can decide and enforce, and the central team is knowledge-bound.
cost Pilot one domain, keep the central core for shared definitions, and accept ongoing duplication as the price of autonomy. Measure ownership by who acknowledged the last alert, not by who is listed (Data Ownership).
Products marketed as "data mesh platforms" are supplying the third commitment — self-service capabilities — and cannot supply the other three, which are decisions about accountability inside your organisation. Evaluate such tools on catalog, contract enforcement, product conformance checks and cross-domain lineage, and verify current capability against documentation rather than against the framing.
How to build it
Most important first.
- Establish the platform first. Domain ownership is only available to organisations whose teams can ship a governed, tested, monitored dataset without building infrastructure. If that is not true today, the platform is the project and the mesh is its consequence (The Self-Service Data Platform).
- Pilot with one domain that has both the strongest data engineering capability and the highest request volume. The first domain is a test of the platform, not of the idea, and it should be chosen to succeed.
- Define the small set of globally shared concepts before distributing anything: the entity identifiers that cross domains, the time grain, and the classification taxonomy. Everything else can be domain-local; those cannot, and they are what make cross-domain joins possible at all (Surrogate Keys).
- Assign the shared entities explicitly to a domain rather than leaving them communal. Customer belongs to whichever team owns the customer lifecycle; if no team will take it, that is important information about whether the organisation is ready (Source of Truth).
- Make governance computational. A policy expressed as a document is applied unevenly across eight teams; the same policy expressed as a platform check that fails a publish is applied identically (Contract Enforcement).
- Move budget, headcount and on-call with the ownership, and verify it: a data product whose owner has never been paged for it does not have an owner (Data Ownership).
- Keep a central core for the definitions that must be identical everywhere — the finance models, the headline metrics. Distributing those buys nothing and costs the one thing centralisation was good at (The Metrics Layer).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Nothing technical. This is an allocation of responsibility, and every guarantee under it is one that a domain team wrote into a product's contract and can be held to (Data Contracts).
- It does guarantee, structurally, that the person who defines a dataset's semantics is the person who knows them — which is the one guarantee a central arrangement cannot make at scale.
- It guarantees nothing about consistency of meaning across domains. That is exactly what federated governance is for, and it works only to the extent that the forum has authority and the platform enforces its decisions (Data Governance).
- It guarantees nothing about quality. A domain team can publish a badly modelled product as easily as a central team can, and now there are more teams able to do so (Data Quality).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that this arrangement specifically needs is cross-domain definition divergence: compute a headline metric from two domains' products and alert when they disagree beyond a stated tolerance. This failure is structurally impossible in a central warehouse and is the characteristic one here (Two Dashboards, Two Numbers).
- Add a product conformance check run by the platform: does every published product declare an owner, a grain, a schema, an SLO, a classification and at least one test? Publishing should fail without them, because a product missing any of the seven is a table (Data Products).
- Both miss a well-formed, well-documented, well-tested product that models its domain incorrectly. Only a consumer who knows the domain catches that, which is why cross-domain review of shared entities is worth the friction (Dataset Documentation).
- Ownership does not make anything fresher. It changes who is able to make it fresher without asking permission, which over a year usually does improve freshness — because the people who feel the pain now control the schedule.
- Freshness becomes per-product and heterogeneous by design. Each product states its own SLO, and consumers joining two products get the worse of the two, which is a fact that must be published rather than discovered (The Freshness SLO).
- The failure to watch for is a product whose stated SLO was written once and never measured. An unmeasured SLO is a sentence in a document (Freshness Monitoring).
- A published data product is a public interface with consumers the owning team may not know. Its schema changes therefore follow the same discipline as an API: additive changes are safe, removals and retypes need deprecation with a date, and consumers need to be enumerable (Backward Compatibility, Impact Analysis).
- The globally shared concepts evolve hardest, because changing one requires every domain to change together. That is precisely why the shared set should be kept as small as it can be while still supporting cross-domain joins.
- The organisation itself evolves — teams merge, split and are reorganised — and data products outlive the teams that created them. Ownership needs a re-assignment procedure, or every reorganisation creates orphans (Data Ownership).
- Recovery is per-product and follows the same rules as anywhere else: immutable raw upstream, deterministic transformations, re-run into a new location and swap (Reprocessing vs Retrying).
- What is harder here is a correction that spans domains. If a shared identifier was wrong for a month, every downstream product in every domain needs reprocessing, and there is no single team that can execute it — so the platform must provide a way to publish an impact notice and track which products have been corrected (Impact Analysis).
- Backing out of the mesh is a recovery nobody plans and some organisations need. Keep the central core models populated throughout so that re-centralising a domain is a re-assignment rather than a rebuild.
What can go wrong
- Ownership assigned without capacity, so datasets have owners who cannot maintain them and quality declines with a name attached.
- Federated governance without authority, producing several defensible definitions of the same business concept and no mechanism to converge them.
- The self-service platform not existing, so each domain rebuilds ingestion and orchestration and the organisation operates several partial platforms.
- Shared entities orphaned, because they belong to every domain and therefore to none (Dimension Tables).
- A proliferation of products that nobody consumes, because publishing is now easy and retiring is still nobody's job (Storage Lifecycle).
- The mitigation failing: a governance forum that responds to divergence by requiring approval for every publish, which restores the central queue with additional ceremony (The Central Warehouse).
- "Data mesh is an architecture." It specifies no storage, no engine, no format and no topology. It is an allocation of ownership and accountability, and it can be implemented inside a single warehouse with a schema per domain (Data Architecture Patterns).
- "Mesh means no central team." It requires a central platform team, and usually a central core of shared definitions as well. What it removes is central *modelling* of every domain, not central capability (Data Platform Engineering).
- "Mesh replaces the warehouse." The warehouse or lakehouse is where the products live. Storage posture and ownership model are separate axes (Lake vs Warehouse vs Lakehouse).
- "We assigned owners, so we have a mesh." Ownership without capacity, budget and on-call is a spreadsheet column. The test is whether the named owner has been paged for the dataset and fixed it (Data Ownership).
- "Federated governance means each team decides." It means a cross-domain forum decides the small shared set and the platform enforces it. Without enforcement it is a mailing list, and the divergence it was created to prevent happens anyway (Contract Enforcement).
- "It will reduce coordination." It reduces coordination *within* the delivery of a dataset and increases it *across* domains. Whether that is a net gain depends entirely on how many of your questions cross a domain boundary.
- Distributing ownership distributes the obligation to classify, mask, retain and delete, which is exactly where it goes wrong without computational enforcement. The policy must be a platform check that fails a publish, not a document each team interprets (Data Classification, Data Access Control).
- Deletion requests are harder in this model than in a central one, because a subject's data exists in products owned by several teams and no single team can enumerate them. Generated lineage across product boundaries is the only workable mechanism (Deletion Requests, Data Lineage).
- Access control becomes federated too: each product declares its access policy, and the platform must apply it consistently, or "self-service" quietly means "everything is readable by everyone" (Row and Column Security).
Operating it
- Products with a named owner who has acknowledged an alert in the last quarter, as a fraction of published products. This is the honest ownership number and it is usually much lower than the owner field suggests (Data Ownership).
- Number of distinct definitions in production for each globally shared concept. Rising is the direct measure of federated governance failing (The Metrics Layer).
- Products meeting their own stated freshness SLO, per product, published where consumers can see it (The Freshness SLO).
- Time from a domain team deciding to publish a product to that product being consumable. If this is measured in weeks, the platform is not self-service and the mesh is not available (The Self-Service Data Platform).
- Consumption per product. Products with zero consumers are cost and maintenance load, and a mesh generates them faster than a central team does (Data Discovery).
- Below roughly a handful of teams the model is pure overhead: the coordination it saves does not exist yet, and the platform investment it requires is unaffordable (The Central Warehouse).
- Its advantage grows with team and source count, not with data volume. An organisation with fifty producing teams gets more from it than one with five hundred terabytes and four teams.
- At high product counts, discovery and lineage become the binding constraint. Nobody can find the authoritative product for a concept, and several claim to be it (The Data Catalog, Data Discovery).
- At high consumer counts, the SLO and contract layer is what keeps it usable, because consumers cannot personally know eight owning teams (Data Contracts).
- Coordination is the dominant cost and it moves rather than disappears: it leaves the central team's backlog and reappears as cross-domain negotiation about identifiers, grains and SLOs (Agreement Costs Round Trips).
- Duplicated effort rises. Several domains will build similar transformations and similar dimension tables, and some duplication is the price of autonomy rather than a failure to be eliminated (Compute Waste).
- Platform investment is a large fixed cost that must be paid before the model is available at all, and it does not scale down — a mesh with a weak platform is more expensive than the central team it replaced (Data Platform Engineering).
- Storage grows faster because publishing is easy and retiring is unowned. Lifecycle policy is not optional under this model (Storage Lifecycle).
- You trade consistency of meaning for throughput of delivery. Both are real goods and no arrangement gives both; the mesh takes the delivery side deliberately and pays for consistency with governance machinery.
- Autonomy means duplication. Several domains solving similar problems slightly differently is the expected steady state, not a defect to be centralised away — centralising it is how the queue comes back.
- The model requires a strong platform team, which means the organisation must invest centrally in order to decentralise. That is counterintuitive and it is the step most commonly skipped.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICEvery element of this model is organisational: who owns a dataset, who is paged, who arbitrates a definition, and who funds the platform. Two companies on identical technology will get completely different results depending on whether ownership came with headcount and on-call or only with a name in a field.
- SCALE-SPECIFICThe model pays off above the point where no central team can hold the semantics of every source — a team-count and source-count threshold rather than a data-volume one. Below it the coordination it saves does not exist and the platform investment it demands is not recoverable.
- GENERALThe four principles are technology-neutral and can be implemented inside a single warehouse with a schema per domain, across separate lakehouses, or over a shared event log. Any claim that a specific product or topology is required by the model is a vendor claim rather than a property of it.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the half of this model that decides whether it works: moving on-call with ownership, paved-road tooling that makes the correct path the easy one, and the delivery discipline a domain team needs before it can be accountable for a production dataset.
- — Distributed Systems owns the analogy this model borrows from — autonomous services with contracts at the boundary — and the ways that autonomy fails when the boundary is under-specified.