Data Platform Engineering
Eight shared capabilities — ingestion, storage, compute, orchestration, catalog, quality, security, observability — and the boundary question that decides whether the platform team is a substrate or a queue.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which part of a data pipeline should a central platform team own, and which part must be owned by the team that knows what the data means?
Two consumers, and they pull in opposite directions. The domain engineer wants to publish a dataset today without learning object-storage layouts, orchestration internals or access-control syntax. The organisation wants every dataset built on that platform to be classified, tested, monitored, discoverable and attributable without asking anyone to remember. A platform that serves only the first produces sprawl; one that serves only the second produces a queue (The Self-Service Data Platform).
The unit here is a capability, not a pipeline. One capability is one thing the platform provides to every team — "scheduled execution with retries and backfill ranges", "a governed publish that applies classification" — with its own interface, its own version and its own failure behaviour. Thinking in pipelines is what turns a platform team into a service desk, because a pipeline belongs to one domain and a capability belongs to all of them.
One central team builds and runs every pipeline. They know the tools, they own the infrastructure, and requests arrive as tickets. For the first several teams this is not merely acceptable, it is optimal — a shared team with real depth beats five teams each learning orchestration badly, and organisations that skip this stage build platforms for problems they do not have yet (The Central Warehouse).
The request queue grows with team count while the platform team grows with budget, and those two numbers stop tracking each other. Lead time from "we need this dataset" to "it exists" goes from days to quarters, and nothing in the platform team's own metrics shows it (Agreement Costs Round Trips).
- The request queue grows with team count while the platform team grows with budget, and those two numbers stop tracking each other. Lead time from "we need this dataset" to "it exists" goes from days to quarters, and nothing in the platform team's own metrics shows it (Agreement Costs Round Trips).
- The platform team writes transformations for domains it does not understand.
order.status = 7gets modelled as "complete" because that is what the ticket said, and the correction arrives from finance three months later (Who Owns Data Quality). - The boundary is drawn at the wrong place: the platform owns the transformation repository, so a domain team changing a business rule needs a platform review. Every business change is now a platform ticket, which is the bottleneck in its purest form.
- The opposite boundary, equally bad: the platform provides an infrastructure module and nothing else, so each domain team builds its own orchestration, its own tests and its own alerting. The organisation now operates six partial platforms and one of them is the good one, by accident (Data Mesh).
- The paved road supports exactly the shape of the pipeline the first customer needed. The second team's source is a SaaS API with cursor pagination, there is no capability for it, and they route around the platform entirely — permanently (Ingestion Sources).
- A platform upgrade changes the default behaviour of every pipeline at once. It is the only deployment in the organisation whose blast radius is every dataset, and it is usually reviewed like a library bump (Rolling Back Data).
What is actually happening
- There are two different scarce resources in a data organisation and they do not live in the same place. Capacity — the ability to run reliable infrastructure — concentrates well; a few people can operate storage, compute and orchestration for everybody. Semantic knowledge — what
order.status = 7means — does not concentrate at all, and every attempt to move it into a central team fails at exactly the point where the number of source systems exceeds what a handful of people can hold (The Central Warehouse). - The boundary follows that split, and the test is a single question: does this decision require knowing what the data means? If it does, it cannot be centralised at any budget. If it does not, centralising it is almost always right, because it is the same work being done differently in six places (Data Ownership).
- A platform therefore provides capability, not labour. The distinction is operational rather than philosophical: labour is consumed by a request and has to be scheduled against other requests, while a capability is available to everyone simultaneously and its cost does not rise with the number of teams using it. A platform team that does work on request has a queue by construction, however good it is (The Self-Service Data Platform).
- Bottlenecks form wherever the platform sits in the critical path of a change rather than underneath it. Reviewing a domain's model, approving a publish, writing the ingestion config — each one is a place where a domain team's lead time contains a platform team's prioritisation, and lead time is what teams actually experience as "the platform".
- The eight capabilities are not a shopping list to be built in order. Each is independently ownable, and the correct sequence is decided by which one is currently the largest thing every team is reinventing — usually orchestration first, catalog and quality much later than they should be (Orchestration, The Data Catalog).
- The platform's deepest leverage is that it is the only place where policy can be made computational. A retention rule written in a document is applied unevenly by eight teams; the same rule expressed as a publish-time check is applied identically, and that difference is worth more than any tool the platform provides (Contract Enforcement, Data Governance).
Eight capabilities, and where the line falls through each one
The capability list is uncontroversial and almost useless on its own, because the interesting content is not *what* the platform provides but *where the ownership boundary cuts through each one*. Every row below is split, and the split is in a different place each time.
The two right-hand columns are the point. A boundary drawn too far right makes the platform a bottleneck: a domain team cannot ship without a platform person, so platform prioritisation becomes organisational lead time. Drawn too far left it makes the platform a hosting provider: everything is possible and nothing is consistent, and governance becomes a series of conversations.
Read the Domain owns column and notice how much of it is meaning — which source is authoritative, what the business rule is, what threshold matters, who may read this. None of that can be centralised at any budget, because centralising it requires the central team to learn every domain, which is the constraint that produced this whole discussion (Data Mesh).
| Capability | Platform owns | Domain owns | Symptom when the platform owns too much | Symptom when it owns too little |
|---|---|---|---|---|
| Ingestion | Connectors, change detection, retry and checkpoint machinery, raw landing, schema capture (The Raw Landing Zone). | Which sources matter, which system is authoritative, what an extract may cost the source (Source of Truth). | Every new source is a platform project, so domains stop asking and start exporting CSVs by hand. | Six bespoke ingestion scripts with six different failure behaviours and no common recovery story. |
| Storage | Layout conventions, table format, file sizing, compaction, lifecycle rules (File Compaction, Storage Lifecycle). | What to retain and for how long, and which partitioning matches the domain's query patterns (Partitioning). | A single mandated layout that suits the first workload and forces full scans for the rest (Partition Pruning). | Small-file sprawl, no lifecycle policy, and a storage bill nobody can attribute (File Size and the Small-Files Problem). |
| Compute | Cluster or warehouse provisioning, isolation between tenants, quotas, autoscaling (Separating Storage from Compute). | Job shape, model complexity, and the cost/freshness trade for its own datasets (Cost vs Freshness). | A capacity request for every workload change, which turns tuning into a ticket. | One team's backfill starving everyone else's scheduled runs, repeatedly (Multi-Tenancy). |
| Orchestration | Scheduling, dependency execution, retries, backfill ranges, idempotent re-runs (Orchestration, Idempotent Data Pipelines). | The DAG itself: what depends on what, and what a correct run of this domain's pipeline means (The Transformation DAG). | Domains submitting DAG changes as pull requests to a platform-owned repository, gated on platform review. | Three orchestrators, no cross-team dependency awareness, and cron jobs holding the gaps together (Scheduler vs Orchestrator). |
| Catalog & lineage | Automatic metadata and lineage capture, search, the registry of published products (Data Lineage, The Data Catalog). | Descriptions, grain statements, semantics and ownership entries — the parts only the domain knows (Dataset Documentation). | A central team writing documentation for datasets it does not understand, producing text that is present and empty. | A catalog populated voluntarily, therefore incomplete, therefore not trusted, therefore not used (Data Discovery). |
| Quality | The test framework, publish gates, alert routing, the quality dashboard (Data Tests, The Data Quality Dashboard). | Which assertions matter, what the thresholds are, and what an anomaly means in this domain (Who Owns Data Quality). | Platform-authored generic tests that fire constantly and are muted within a month (Alert Fatigue: The Page Nobody Reads). | Every domain inventing its own testing approach, with no shared way to see whether anything is tested at all. |
| Security & governance | Identity, grants, masking, retention enforcement, audit trails — policy as executable checks (Data Access Control, Data Masking, Tokenisation & Encryption). | Classification of its own fields and approval of who may read them (Data Classification). | A central approval step on every access request, which teaches people to ask for the widest grant once (Least Privilege). | Warehouse defaults as the access policy, which usually means everyone can read everything (Row and Column Security). |
| Observability | Freshness, volume and schema monitoring by default for every dataset on the paved road (Data Observability, Freshness Monitoring). | The SLO it promises consumers, and being paged when its own dataset breaks (The Freshness SLO). | The platform on-call receiving alerts about domains' data, which they cannot diagnose and will eventually route to nobody. | Datasets with no monitoring at all, where every incident is reported by a person reading a dashboard (Stale Dashboards). |
Eight rows, one pattern: the platform owns how, the domain owns what and why. Every bottleneck in the fourth column is a case of the platform being asked to supply meaning, and every gap in the fifth is a case of mechanism being reinvented per team.
Capability, not labour
The single distinction that decides whether a platform team scales is whether its output is work done for you or something you can use. Labour is consumed by a request; it has to be prioritised against other requests, and its supply is fixed at the size of the team. A capability is available to every team at once, and its cost does not rise when a ninth team starts using it.
That is why the ticket queue is not a symptom of an under-resourced platform team. It is a structural property of a platform that produces labour, and hiring makes the queue shorter without making it a different kind of thing. The escape is to convert the most-requested labour into an interface — which is a different activity from doing the work faster (The Self-Service Data Platform).
The diagram below draws the boundary as a substrate rather than a stage in the flow. Everything above the line is domain-owned and changes constantly; everything below is platform-owned and changes slowly and deliberately. The dashed edges are enforcement: the platform is not in the path of a domain's *decisions*, but it is in the path of every domain's *publish*, which is precisely where governance can be applied without slowing anyone down (Contract Enforcement).
Domain teams describe what they need in a ticket. The platform team builds the ingestion, writes the transformation, sets the schedule and adds the tests. Requests are prioritised quarterly, and the platform team is paged when a domain's numbers look wrong.
Domain teams declare a source, a model, an owner, tests and a classification. The platform generates the pipeline, captures lineage, applies policy at the publish gate, monitors the result by default, and pages the declared owner when it breaks. The platform team's work is capabilities and defaults, not pipelines.
Labour is rivalrous and capability is not. A team producing labour has a queue whose length grows with the number of consumers, so its lead time is set by its own capacity — and that capacity has to grow linearly with the organisation to hold lead time constant, which no budget does. A team producing capability serves a ninth consumer at nearly no marginal cost, and the enforcement it applies at the publish boundary is uniform precisely because nobody had to remember to ask for it (Agreement Costs Round Trips).
How a platform team becomes the bottleneck
None of these failures looks like a failure while it is happening. The platform team is busy, the tickets are being worked, the infrastructure is up, and the dashboards are green — and the organisation's lead time from idea to dataset is quietly measured in quarters (Data Platform Anti-Patterns).
The cause column has one repeated shape: the platform ended up in the critical path of a decision that requires domain knowledge. That is the diagnostic. When the answer to "why is this waiting" is "the platform team has to look at it", ask whether the thing being looked at is mechanism or meaning — and if it is meaning, the boundary is in the wrong place and no amount of hiring fixes it.
The last row is the honest counterweight. A platform that responds to all of this by removing every gate does not become faster; it becomes a hosting provider, and the governance that was the reason for centralising anything evaporates. The target is a platform that is easy to use and hard to bypass, not one that is merely easy (Data Governance).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A domain needs a new source ingested. | A ticket sits for a quarter; the domain builds a manual export in the meantime and it becomes permanent. | Ingestion is provided as labour rather than as a capability, so each new source consumes platform capacity (Ingestion Sources). | Turn the most common source shapes into declarative connectors, and publish the interface for adding a new one. Measure how many sources were onboarded without platform involvement. |
| A business rule changes in one domain. | The change waits for a platform code review by someone who does not know the rule. | The platform owns the transformation repository, so domain logic sits inside a platform-owned artefact (The Transformation DAG). | Move the models into domain-owned repositories, keep the framework and the gates central, and review contracts rather than logic (Data Contracts). |
| A second team tries to use the paved road. | Their source or schedule does not fit; they build their own pipeline outside the platform and nobody notices for a year. | The paved road encodes the first customer's shape as a universal one. | Design capabilities against at least two dissimilar consumers before generalising, and treat paved-road coverage as the metric that catches this (The Data Catalog). |
| Access to a governed dataset is requested. | Approval takes days; requesters learn to ask for the broadest possible grant once, and keep it. | The platform is in the approval path for a decision the data owner should make (Data Access Control). | Platform provides the mechanism and routes approval to the declared owner, with time-bound grants as the default (Least Privilege). |
| A platform release changes a default. | Several domains' datasets change behaviour overnight and the incident is diagnosed as a data problem. | A platform change has organisation-wide blast radius and was reviewed like a library upgrade (Semantic Changes). | Version defaults, roll out per tenant, and publish the affected-pipeline list before the change rather than after (Impact Analysis). |
| The platform team responds to complaints by removing gates. | Lead time improves and, a year later, nothing is classified, half the datasets are unowned, and cost is unattributable. | Gates were treated as friction rather than as the platform's reason to exist — the mitigation failing (Data Governance). | Keep the gates and remove the *waiting*: enforcement should be automatic and instantaneous at publish, never a human in a queue (Contract Enforcement). |
What should the platform take on next?
A platform is built one capability at a time, and the sequence matters more than the list. The right next capability is the one that most teams are currently reinventing badly, which is a question about your organisation rather than about platform engineering.
The options below are ordered by how often they turn out to be the answer, not by importance. Orchestration and raw landing are usually first because they are pure mechanism with no domain knowledge in them at all. Catalog and lineage are usually late and usually should have been earlier, because their value is retrospective — they are only useful if they were capturing metadata before the incident that needed them (Data Lineage).
Whatever is chosen, apply the same test to it before building: does using this capability require the platform team to know what the data means? If yes, you are about to build a queue. If no, you are about to remove one.
What are teams currently reinventing, and does centralising it require domain knowledge?
when Teams are running pipelines from cron, notebooks or CI, with no dependency awareness and no shared retry or backfill behaviour.
cost The platform now owns the availability of everyone's pipelines and inherits a correlated failure mode. Buys idempotent re-runs, backfill ranges and dependency execution for every team at once (Orchestration).
when Teams transform on ingest, keep no raw history, and cannot reprocess after finding a bug (Keeping Raw History: The Recovery Position and the Liability).
cost Storage grows and someone must own lifecycle policy. Buys organisation-wide recoverability, which is the capability that turns most mistakes into re-runs (Reprocessing vs Retrying).
when Datasets reach consumers with no owner, no tests and no classification, and governance is a periodic review.
cost Publishing gets slower and some teams resist. Buys policy applied identically everywhere, which is the only version of governance that survives growth (Contract Enforcement).
when Nobody can answer "what feeds this", "what else does this feed", or "who is this bill for" (Impact Analysis).
cost Requires instrumenting every pipeline, and its value arrives later than its cost. Buys impact analysis, deletion enforcement and cost accountability — none of which can be reconstructed retroactively (Cost Attribution).
when Data incidents are reported by humans reading dashboards rather than by monitors (Data Observability).
cost Alert routing must reach domain owners rather than the platform, or the platform absorbs on-call for data it does not understand (Alert Fatigue: The Page Nobody Reads).
when The capabilities exist and are good, and the constraint is that domain teams have nobody able to use them (Who Owns Data Quality).
cost Headcount rather than platform work, and a period where the platform looks under-invested. Buys the only thing a platform cannot provide, which is people who know what the data means.
Managed platforms and "data platform in a box" products supply capabilities — orchestration, catalog, quality tooling, access control — and cannot supply the boundary. Which capability a given product covers, and how completely, changes release by release, so evaluate against the eight-row table above and re-check current documentation rather than the marketing category. The decision that determines whether your platform team is a substrate or a queue is not affected by any of these purchases.
How to build it
Most important first.
- Draw the line by knowledge, not by technology. The platform owns mechanism — how work runs, where bytes live, how policy is applied. Domains own meaning — what the data represents, what the business rule is, what the SLO should be. Write that sentence down, because every boundary dispute is resolved by it (Data Products).
- Build a paved road with visible escape hatches. The paved path should be so much easier that teams choose it, and the escape hatch should exist so that the one team with a genuinely different problem does not have to route around the entire platform silently (Data Architecture Patterns).
- Ship capabilities as declarative interfaces and libraries rather than as services teams file requests against. The measure of a capability is whether a team can use it at 22:00 on a Friday without anyone from the platform team being awake (The Self-Service Data Platform).
- Give the platform its own SLOs, published to its consumers: availability of scheduled execution, lag of the catalog and lineage graph, time to onboard a new source type. A platform without SLOs to its users is a platform that will be judged by anecdote (Pipeline SLOs).
- Never own transformations that encode business rules. The platform can own the *framework* that runs them, tests them, versions them and monitors them — and every line of domain logic inside it belongs to the domain (The Transformation DAG).
- Own the boundary enforcement completely: contract checks, classification, publish gates, lineage capture, cost attribution. These are exactly the things that are invariant across domains and worthless if applied inconsistently (Contract Enforcement, Cost Attribution). Measure lead time from a domain engineer's intent to a governed dataset in production, and treat it as the platform's primary metric. Every other platform metric can look excellent while that number is six weeks (The Self-Service Data Platform).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The platform guarantees that a pipeline built on the paved road gets scheduled execution, retries, lineage capture, classification enforcement and monitoring by default rather than by remembering. That default is the product (Data Observability).
- It guarantees availability and behaviour of the capabilities it publishes, at whatever SLO it states. It guarantees nothing about the correctness of any dataset built with them — the platform cannot know what the data means, which is the entire reason the boundary exists (The Pipeline Succeeded. The Data Is Wrong.).
- It guarantees policy is applied uniformly to everything on the paved road, and explicitly nothing about work that routed around it. Shadow pipelines are outside every guarantee the platform makes, and their existence is a platform metric rather than a domain problem (Data Governance).
- It does not guarantee that a capability fits every domain. A paved road is a claim about the common case, and a platform that claims universality is one that will be routed around quietly.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that tells you most about a platform is paved-road coverage: what fraction of datasets serving production consumers were created through the platform path, discovered by comparing the warehouse's own inventory against the platform's registry. Everything not on the list is ungoverned, unmonitored and unattributed (The Data Catalog).
- It misses whether the paved-road datasets are any good. A perfectly governed pipeline can produce nonsense, and the platform is structurally unable to tell — it has no idea what the numbers mean (Data Quality).
- Pair it with lead time to a first published dataset, measured for real teams rather than in a demo. Coverage measures compliance; lead time measures whether the platform is a substrate or a queue, and only the second one predicts whether coverage will keep rising (The Self-Service Data Platform).
- The platform decides the range of freshness available to any dataset built on it. If the only ingestion capability is a nightly batch extract, no domain can promise an hourly SLO regardless of how much its consumers want one (Batch vs Streaming Ingestion).
- It also owns the freshness of its own metadata. A catalog whose lineage graph is a day behind is a catalog that is wrong during exactly the incident where it is needed, and lineage lag is a platform SLO that almost nobody states (Data Lineage).
- The platform's scheduling capacity is shared, so one domain's enormous backfill becomes another domain's missed freshness SLO unless the budgets are isolated. Isolation between tenants is a freshness decision wearing an infrastructure costume (Multi-Tenancy).
- A platform capability is an interface with consumers who cannot all move at once, so it evolves like a public API: additive changes are safe, behaviour changes need a version, and removals need a deprecation window with a date (Backward Compatibility).
- The dangerous change is one that alters a default, because defaults are what most pipelines run on. Changing a default retry policy, a default file layout or a default publish mode is a change to every dataset simultaneously, and it has no schema footprint anywhere (Semantic Changes).
- Capability migrations need the expand-contract shape: run the new capability beside the old, migrate teams individually, and remove the old one only when the registry says nobody is on it. A platform that cannot enumerate its users cannot deprecate anything (Impact Analysis).
- Platform failures are correlated failures: when scheduled execution is down, every domain's pipeline is down together, and the recovery is a single coordinated catch-up rather than eight independent ones. The platform therefore needs backfill capability as a first-class feature, not as a script each team writes (Backfills).
- A bad platform release has to be rolled back at the platform level while domain pipelines are mid-run, which is the one operation nobody rehearses. Rehearse it: the alternative is discovering the semantics of a partial rollback during an incident (Rolling Back Data).
- Rebuilding a domain's datasets after a platform incident is only possible if raw landing is a platform capability rather than a per-team convention. Centralising the raw layer is what makes organisation-wide recovery available at all (The Raw Landing Zone, Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- The platform team as a ticket queue, where lead time grows with team count and every domain's roadmap contains a platform dependency (Agreement Costs Round Trips).
- The platform owning transformations, so business-rule changes require people who do not know the business (Who Owns Data Quality).
- A paved road shaped around the first customer, which the second and third customers cannot use and quietly abandon.
- Shadow platforms: teams routing around the paved road with their own schedulers and their own storage, invisible to governance and to cost attribution (Data Platform Anti-Patterns).
- A default changed in a platform release, altering the behaviour of every pipeline at once with no dataset-level review (Rolling Back Data).
- The mitigation failing: an escape hatch so convenient that it becomes the main road, at which point the platform provides infrastructure and no guarantees at all.
- "The platform team owns data quality." It owns the *machinery* of quality — the test framework, the gates, the alerting. Whether a threshold is right and whether a rule matches the business is a domain question, and assigning it centrally guarantees it is answered badly (Who Owns Data Quality).
- "A data platform is a set of tools." Tools are the least durable part. The platform is the set of defaults, boundaries and enforcement points that make the correct path the easy one; the same tools produce a swamp in an organisation without them (Data Platform Anti-Patterns).
- "Self-service means we do not need a platform team." It means the platform team builds capabilities instead of pipelines. The team gets more important, not less — it is now on the critical path of every team indirectly rather than one team directly (The Self-Service Data Platform).
- "Standardise on one engine and the platform is done." Engine choice is a small part of a platform and the part most likely to change. Catalog, contracts, classification, lineage and cost attribution outlive every engine decision you will make (Query Engines).
- "The platform should build the pipelines — they are the experts." They are experts in the mechanism and strangers to the meaning. That arrangement works until the number of source systems exceeds what one team can hold, and then it fails silently rather than loudly (The Central Warehouse).
- The platform is where governance stops being a document and becomes a mechanism. Classification, masking, retention and access are applied uniformly only if they are applied by a shared publish path (Data Classification, Data Access Control).
- Everything routing around the paved road is outside every one of those controls, which makes paved-road coverage a governance metric rather than an engineering vanity number (Data Governance).
- Deletion requests are executable across an organisation only if lineage was captured centrally as datasets were created. A platform that captures lineage after the fact captures an incomplete graph, and an incomplete deletion is a failed deletion (Deletion Requests, Data Lineage).
Operating it
- Lead time from a domain engineer starting work to a governed dataset serving a consumer, measured per team. This is the platform's primary signal and it is the one most platform dashboards do not have (The Self-Service Data Platform).
- Paved-road coverage: production datasets created through the platform, as a share of production datasets that exist. A falling number means teams are routing around, and it falls long before anyone says so (The Data Catalog).
- Open platform requests, and their age distribution rather than their count. A stable queue of increasingly old requests is the signature of a capability that should exist and does not.
- Blast radius of the last platform change: how many pipelines ran with a changed default. If nobody can compute that number, platform changes are being deployed blind (Impact Analysis).
- Split of platform on-call incidents into "the platform broke" and "a domain's data broke". A platform team paged for the second kind is a platform team that has absorbed ownership it cannot discharge (Data Ownership).
- Cost attributed to each consuming team rather than pooled centrally, because an unattributed platform cost is one nobody has any reason to reduce (Cost Attribution).
- Below a handful of teams, the platform team *is* the data team and should be. Building capabilities for tenants who do not exist yet is the most common way to spend a year producing nothing (The Central Warehouse).
- The line moves as the organisation grows: work that was correctly centralised at five teams is a bottleneck at twenty-five, and the migration is from *doing* to *providing*. Recognising that moment is the core judgement of the role.
- At high team counts, everything the platform does by hand must become an interface, because the platform team's capacity is fixed while demand is linear in team count (The Self-Service Data Platform).
- At high dataset counts, catalog, lineage and cost attribution stop being nice-to-haves and become the only way anyone can find, trust or afford anything. They are also the capabilities that are hardest to add late, because they need to have been capturing metadata all along (Metadata: Technical, Operational and Business, Data Discovery).
- A platform is a large fixed cost that does not scale down. It is paid before the first domain team benefits, and the return arrives only as the number of teams using it grows — which is why platform investment in a small organisation is genuinely wasteful and in a large one is the cheapest thing available (What Actually Drives Data Platform Cost).
- The largest hidden cost of a badly-drawn boundary is not the platform team's time; it is the waiting time of everyone else. A queue converts one team's capacity limit into an organisation-wide lead time, and that cost never appears on the platform's own ledger (Agreement Costs Round Trips).
- Shadow platforms cost twice: duplicated infrastructure, plus the governance work that has to be done manually for everything outside the paved road (Compute Waste).
- Pooled platform cost with no attribution guarantees over-consumption, because every team's marginal cost of another daily full refresh is zero to them and real to the organisation (Cost Attribution, Full Refresh vs Incremental).
- Standardisation buys uniform governance, shared expertise and one thing to operate; it costs fit, and every domain with an unusual source pays for the standard in workarounds. The paved road is that trade made deliberately rather than by accident.
- Escape hatches keep the platform honest and dilute its guarantees. A platform with none is routed around silently; a platform with one that is too comfortable is an infrastructure team with a nicer name.
- Investing centrally in order to decentralise is counter-intuitive and it is the actual shape of the work. The platform has to be strong *before* domain ownership is available, which means the payoff arrives a long way after the spend (Data Mesh).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICWhere the boundary belongs is a function of how many teams there are, how capable they are, and whether the organisation will fund a team whose output is other teams' throughput. The same capability set produces a substrate in one company and a queue in another, and the difference is entirely in whether the platform does work on request.
- SCALE-SPECIFICWith a handful of teams a central team that builds every pipeline is correct and a platform of capabilities is premature. The inversion happens when request lead time starts growing with team count — a threshold in team count and source-system count rather than in data volume.
- GENERALThe eight capabilities and the knowledge-versus-mechanism test are independent of stack: they apply to a single managed warehouse, to a lakehouse on object storage, and to a streaming platform. What changes is which capabilities you buy rather than build, not which ones have to exist somewhere.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns most of what a platform team does with software: paved roads, internal developer platforms, CI, infrastructure as code, progressive rollout and the delivery discipline that makes a platform change safe. This lesson is about the data-specific boundary; that domain owns the craft of providing a platform at all.
- — Distributed Systems owns the correlated-failure reasoning that a shared platform forces: when one scheduler is down every pipeline is down, and the recovery is a single coordinated catch-up whose ordering constraints are not obvious.