The Central Warehouse
The arrangement most organisations actually run, taken seriously: one team, one place, one definition — with a real advantage and a specific failure mode that arrives with source count rather than with data volume.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
One team ingests everything, models everything and serves everyone. What does that genuinely buy, and at what point does it stop working?
Analysts and finance, who get a single credential, a single schema to learn and a single answer to "what is revenue". Also every executive who has ever received two numbers for the same quarter and had no way to decide which was right (Two Dashboards, Two Numbers).
The unit is a modelled table owned by the central team — one row per business event or entity at a declared grain, produced by transformations that team wrote and maintains. The important property is not where the bytes live; it is that one group of people can answer "what does this column mean" for every column in the platform (Grain: What Does One Row Represent?).
Hire a data team, point it at every source, and let it build the warehouse. This is not a strawman — it is the correct first architecture for almost every organisation, it produces value within weeks, and the alternatives all require infrastructure that does not exist yet.
The team becomes a queue. Every new dataset, every column addition and every metric change is a ticket, and the backlog grows faster than the team, because each delivered dataset creates its own maintenance load (Data Platform Anti-Patterns).
- The team becomes a queue. Every new dataset, every column addition and every metric change is a ticket, and the backlog grows faster than the team, because each delivered dataset creates its own maintenance load (Data Platform Anti-Patterns).
- Domain knowledge thins. At five source systems one team can hold what every field means. At fifty it cannot, and it starts modelling from column names — which produces tables that are structurally correct and semantically wrong (Semantic Changes).
- Shadow pipelines appear. Teams that cannot wait build their own extracts, and now there is a second, ungoverned copy of the data with no owner, no tests and no lineage — which was the exact problem the central warehouse was supposed to prevent (Data Discovery).
- Upstream changes arrive unannounced. The central team is not in the producing team's planning, so a rename lands as a broken dashboard rather than as a deprecation notice (Breaking Schema Changes).
- Definitions ossify.
active_customerwas defined in year one, three hundred dashboards depend on it, and no single team is willing to own the migration — so the definition survives long after it stopped describing the business. - Prioritisation becomes political. The central team's roadmap is a scarce resource allocated across the whole company, and the loudest requester wins rather than the highest-value one.
What is actually happening
- The advantage is real and it is worth naming precisely: collocating the modelling with the governance and the tooling gives one enforcement point. One place to apply access control, one place to define a metric, one team who can trace any number to its source (Data Lineage).
- That advantage comes from the same property as the failure: all of the meaning lives in one team. Meaning does not compress. Every additional source system adds domain knowledge that must be held by someone, and a central team's capacity to hold it is roughly fixed.
- So the scaling limit is source count and consumer count, not bytes. A central warehouse handling petabytes from four systems is comfortable. One handling gigabytes from ninety systems, each owned by a different team, is not — and adding engineers does not fix it, because the coordination cost between them grows superlinearly (Agreement Costs Round Trips).
- The failure is usually misdiagnosed as a capacity problem, because it presents as a backlog. Capacity is the visible symptom; the cause is that a request cannot be served without domain knowledge the central team does not have and cannot cheaply acquire (Data Ownership).
- There are two separable things being centralised: the platform (ingestion, compute, orchestration, catalog) and the modelling (what the tables mean). Centralising the platform scales indefinitely and is generally correct. Centralising the modelling is what hits the wall (Data Platform Engineering).
What one team in one place genuinely buys
The central warehouse is usually introduced as the thing newer patterns exist to escape, which makes it hard to see what it is good at. It is good at exactly one thing, and that thing is the hardest problem in this domain: agreement about what data means.
When one team models every source, customer_id means one thing, revenue is computed one way, and the join between orders and payments was written once by someone who understood both. When several teams model independently, each of those becomes a negotiation, and the negotiation has to happen before anyone can answer a cross-domain question. The central warehouse pays that coordination cost once, at modelling time, instead of every time a question crosses a boundary.
It also collapses several operational concerns into one place. Access control is administered once. Lineage is complete because the same team wrote every hop. An incident has one on-call rota. None of these are glamorous, and together they are why this arrangement survives contact with reality far better than its reputation suggests.
- 1Request arrives
A team asks for a dataset, a column or a metric change.
guarantees That it will be prioritised against every other request in the company, by a team with full context on the platform and partial context on the domain.
fails by Waiting. The queue is the arrangement's defining cost and its most visible one.
- 2Source understanding
The central engineer works out what the source fields mean.
guarantees Nothing. This is the step that has no technical support and no guarantee attached, and it is where the arrangement actually breaks.
fails by Modelling from column names when the producing team is unavailable, producing a structurally valid and semantically wrong table.
- 3Ingestion
Extracts or subscribes to the source and lands it raw.
guarantees At-least-once arrival of what the extract predicate captured. Not completeness (Incremental Extraction).
fails by An extract window that misses late-committing rows, discovered weeks later by reconciliation.
- 4Central modelling
Transforms raw into shared facts and dimensions at declared grains.
guarantees One definition per concept inside the platform, and lineage that is complete because one team wrote it.
fails by A join at the wrong grain, or a definition that made sense for the first consumer and not the fifth (Grain: What Does One Row Represent?).
- 5Validation
Runs the platform's standard test suite before publishing.
guarantees That the assertions the central team wrote hold. Never that the number means what the requester wanted.
fails by Passing every test while the metric is semantically wrong, because no test encoded the domain meaning (Data Tests).
- 6Serving
Publishes to a governed table with access policy applied.
guarantees Atomic publish, one enforcement point for access, one place to look during an incident (Atomic Publish).
fails by Being bypassed. Every consumer who read the source directly is outside every guarantee in this column.
Read the second stage. It is the only one with no guarantee and no tooling, and it is the one that decides whether the output is right. Every other stage in this table has an engineering answer; that one has an organisational answer, which is what the rest of this module is about.
How it degrades, and what the symptom looks like first
The central warehouse does not fail suddenly. It degrades along a predictable path, and every stage of that path is initially read as something else — a hiring problem, a prioritisation problem, a tooling problem. Recognising the pattern early is worth more than any specific mitigation, because the effective responses all take quarters.
The tell that distinguishes structural degradation from a temporary backlog is the fraction of requests that stall on a question about meaning rather than on engineering effort. If most tickets are blocked waiting for someone in another team to explain what a field represents, the constraint has moved from capacity to knowledge, and no amount of hiring into the central team addresses it.
The response column below is deliberately unglamorous. Producer-side contracts, a shadow-pipeline census and a paved road for the highest-volume requesters buy years, and each is cheaper than an architecture migration. Reach for the ownership axis when those have been tried and the constraint is still binding (Data Mesh).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Request volume exceeds the team's throughput. | Backlog grows; requesters start estimating in quarters. | Every delivered dataset adds maintenance load, so team capacity for new work grows more slowly than the request rate. | Publish the queue and its age. Retire unused datasets aggressively — the maintenance load of a dataset nobody reads is pure drag (Storage Lifecycle). |
| Source count grows past what one team can hold. | Models are delivered on time and are subtly wrong; consumers stop trusting new tables. | Semantic knowledge is not compressible. The engineer models the columns because nobody explained the concept (Semantic Changes). | Require a domain reviewer to sign the model definition before publish. It is a slower path and it is the only one that catches this class of error. |
| A team cannot wait for the queue. | An unexplained rise in load on a source database; a spreadsheet with numbers nobody can trace. | A shadow pipeline. Waiting has a cost and teams will pay around it (Data Platform Anti-Patterns). | Census the direct connections, then offer a supported path rather than a prohibition. Prohibition without an alternative just makes the shadow pipeline harder to find. |
| Producer renames or retypes a field. | A dashboard reports zero or nulls, days later, discovered by a human. | The central team is downstream of a change process it is not part of, so it learns about changes by breaking (Breaking Schema Changes). | A contract at the boundary with a check that fails the producer's deploy, not just the consumer's pipeline (Contract Enforcement). |
| A core definition needs to change. | Everyone agrees it should change; the change never happens. | Blast radius spans the company and the owning team lacks the authority to make consumers migrate. | Expand and contract: publish the new definition alongside the old with both documented, migrate consumers by name, then remove the old one on an announced date (Impact Analysis). |
| The paved road still requires central review of every change. | Teams describe self-service as slower than filing a ticket. | The mitigation failed: the review step reintroduced the queue with extra ceremony and worse ergonomics. | Move the review into automated policy — required owner, required tests, required classification — and reserve human review for changes that cross a domain boundary (The Self-Service Data Platform). |
Centralising the platform is not centralising the meaning
The most useful distinction in this lesson is between two things that are usually adopted together and are separable. Platform centralisation means one team owns ingestion, compute, orchestration, catalog, testing and observability. Model centralisation means one team owns what the tables mean.
Platform centralisation scales well and is almost always correct: shared capabilities have real economies, and the alternative is every team maintaining its own orchestrator. Model centralisation is the half that hits the knowledge ceiling. Organisations that conclude "central did not work for us" have frequently abandoned both, and then discovered that giving each domain team its own ingestion stack was worse than the queue they escaped.
This is why the two comparisons below are worth keeping separate in any architecture discussion. The first is a genuine trade with a defensible answer either way. The second is much closer to a settled question, and the answer is that shared capabilities belong to a platform team whatever the modelling arrangement is (Data Platform Engineering).
What is currently blocking the organisation from answering the questions it needs to answer?
when Requests are served in acceptable time and delivered models are trusted.
cost None. Keep it, keep the raw layer immutable, and watch request age and the shadow-pipeline census as leading indicators.
when Tickets are held up by work rather than by questions about meaning.
cost A capacity and prioritisation problem. More engineers, better tooling and retiring unused datasets all help, and none of them require an architecture change.
when Tickets stall waiting for a producing team to explain what a field represents.
cost The knowledge ceiling. Start with domain reviewers on model definitions and producer-side contracts; consider moving ownership for the domains with the most traffic (Data Mesh).
when Most analytical reads bypass the platform entirely.
cost Centralisation is already nominal. The choice is to make the governed path faster than the shadow path, or to formalise domain ownership so the shadow pipelines become supported products (Data Products).
when Analytics throughput has become a named business constraint.
cost The ownership axis is the only remaining lever, and it costs data engineering capacity in domain teams plus a governance forum with authority. It is not available until the platform is genuinely self-service (The Self-Service Data Platform).
Decide "central versus decentralised" once, and apply the answer to both the platform and the modelling. Adopting distributed ownership therefore means dissolving the platform team and letting every domain assemble its own ingestion, orchestration and testing.
Keep the platform central — one ingestion framework, one orchestrator, one catalog, one test harness, one observability surface. Decide model ownership on its own axis, based on whether the central team can still hold the semantics of every source.
The two decisions have different scaling limits. Shared capabilities get cheaper per team as team count rises, because the fixed cost of building an orchestrator is amortised. Semantic knowledge gets more expensive per source, because it does not compress and cannot be shared by writing more code. Bundling them forces one answer onto two curves that bend in opposite directions.
How to build it
Most important first.
- Run it. For most organisations this is the right architecture and the alternatives are premature. The failure mode described here arrives at a specific scale, and building for it early costs more than it saves (From a Simple App to a Scaled System).
- Separate platform ownership from model ownership from the start, even while both sit in the same team. That boundary is what makes a later split a re-assignment rather than a rewrite.
- Make the source teams accountable for the schema at the boundary, not just the central team accountable for absorbing changes. A contract with a producer is the only mechanism that turns an unannounced rename into a negotiated one (Data Contracts).
- Publish the definitions where consumers read the numbers, not in a wiki. A metric layer that computes from one definition prevents the divergence that a documented definition merely describes (The Metrics Layer).
- Track the queue explicitly: request age, delivered datasets per quarter, and the fraction of requests that needed knowledge the team did not have. Those three numbers tell you when the arrangement is reaching its limit, months before anyone argues about it.
- Give the highest-volume requesters a supported path to serve themselves within the platform — a reviewed model directory, a shared testing framework, a paved road — rather than letting them build shadow pipelines outside it (The Self-Service Data Platform).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- One definition per metric inside the warehouse boundary. It guarantees nothing about extracts, spreadsheets, notebooks or the BI tool's own derived fields, and those are where most divergence actually lives.
- A single enforcement point for access control and classification, which is a genuine and underrated guarantee — it is much easier to prove who can see a column when there is one system to ask (Data Access Control).
- Traceability: someone can answer "where did this number come from" for every table, because the same team wrote all of them (Where Did This Number Come From?).
- It guarantees nothing about freshness, completeness or correctness. Those are properties of ingestion and validation, and a central warehouse can be as wrong as any other arrangement — it is just wrong in one place (The Pipeline Succeeded. The Data Is Wrong.).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check this arrangement makes uniquely cheap is cross-source reconciliation: because one team holds every source, it can compare a metric computed from the operational database against the same metric computed from the payment provider and alert on divergence (Reconciliation).
- The check it makes uniquely necessary is a shadow-pipeline census: scan for direct connections to source systems and for scheduled exports that do not originate in the platform. Every one found is an ungoverned copy, and their count is the honest measure of whether centralisation is actually holding.
- Both miss the semantic failure that is characteristic here: a central engineer modelling a field they do not understand. Nothing technical catches that. Only a consumer who knows the domain will, which is the argument for putting the model review in front of them (Dataset Documentation).
- Central modelling does not make anything slower by itself. What creates the perception is queue latency: the data is a day old and the *change* to the model is a quarter away, and consumers experience both as staleness.
- A single team also means a single set of schedules, which tends to converge on one cadence for everything. That is efficient and it means the dataset that genuinely needed hourly is on the nightly run because that is what the platform does.
- The honest freshness statement under this arrangement is per-dataset and published, exactly as under any other. A platform-level freshness number hides the one table that stopped updating on Friday (Freshness Monitoring).
- Upstream schema changes are absorbed centrally, which is efficient and fragile: it works while the team has capacity, and every change absorbed silently is one the producer never learned to announce (Schema Evolution).
- Changing a widely-used definition is the hardest evolution in this arrangement, because the blast radius is the whole company and the owner is a team without the authority to force consumers to migrate. Expand-and-contract applies: publish the new definition alongside the old, migrate consumers, then remove (Impact Analysis).
- Evolving out of the arrangement itself is a re-assignment of ownership. Move one domain at a time, keep the central tables serving through the transition, and treat the first domain as a test of whether the platform is genuinely self-service (Data Mesh).
- Recovery is comparatively easy here, and it is the arrangement's quietest advantage: one team knows how everything was built, so a rebuild from raw is a known procedure rather than an archaeology project (Reprocessing vs Retrying).
- Keep raw immutable and outside the modelled layer. Every recovery story in this arrangement depends on it, including the recovery from a modelling mistake that has been wrong for six months (Keeping Raw History: The Recovery Position and the Liability).
- The recovery this arrangement is bad at is recovering from the *departure of the people*. Meaning held in one team is meaning that leaves with it, and documentation is the only mitigation — which is why it should be a deliverable rather than an intention (Dataset Documentation).
What can go wrong
- The backlog grows monotonically and the team is judged on delivery speed rather than on the structural reason it is slowing.
- A model is built from column names because nobody with domain knowledge reviewed it, and the resulting metric is confidently wrong (Grain: What Does One Row Represent?).
- Shadow pipelines proliferate and the platform quietly stops being the source of truth without anyone declaring it (Source of Truth).
- The team becomes a single point of failure for the company's ability to answer questions, and hiring cannot fix it fast enough because onboarding requires the domain knowledge that is the scarce thing.
- The mitigation failing: a "self-service" model directory that still requires a central engineer to review, deploy and debug every change is a ticket queue with extra ceremony (The Self-Service Data Platform).
- "Central warehouses do not scale." They scale to very large data with a small team. What does not scale is one team holding the meaning of an unbounded number of source systems, which is a knowledge limit rather than a technical one.
- "The backlog means we need more data engineers." Sometimes. More often it means requests require domain knowledge the team cannot cheaply acquire, and another engineer joins the same queue rather than shortening it.
- "Everything must live in the warehouse." Unstructured sources, ML feature source data and raw archives need a landing zone regardless. If it is not named, you have an ungoverned one (The Raw Landing Zone).
- "Centralisation guarantees one definition." It guarantees one definition of the tables the platform owns. The BI tool's calculated fields, the finance spreadsheet and the analyst's notebook are outside that boundary and are where divergence usually starts.
- "This is the legacy pattern." It is the current architecture of a very large number of successful companies, and for most of them it is the correct one. The pattern to be suspicious of is the one adopted before its constraint appeared.
- A single enforcement point is this arrangement's strongest governance property: one place to apply row and column policies, one audit trail, one classification taxonomy (Row and Column Security).
- That property is only real to the extent that consumers actually read through the platform. Every shadow extract is a governance gap that inherits the data's obligations and none of its controls (PII in Pipelines).
- Deletion requests are comparatively tractable here, because one team can enumerate every derived copy — provided the lineage is generated rather than remembered (Deletion Requests).
Operating it
- Median and ninety-fifth-percentile age of open data requests. The ninety-fifth percentile is the one that predicts shadow pipelines (Percentiles: Which One, and How Many Users Is That?).
- Count of direct connections to source systems from outside the platform, sampled from database logs. It is the census of what centralisation has already lost.
- Number of models with no test and no documented owner within the central layer. Centralisation does not prevent orphans; it just means they all belong to one team (Data Ownership).
- Fraction of incidents whose root cause was an unannounced upstream change. A high fraction is evidence that contracts are missing at the boundary, not that the team is careless (Data Incidents).
- At 10x data volume: nothing about the arrangement changes. This is layout and incremental processing work, and it is orthogonal (Incremental Processing).
- At 10x source count: the arrangement is under real strain. No central team holds the semantics of ninety systems, and the failure appears as quietly wrong models rather than as missed deadlines.
- At 10x consumer count: discoverability and access administration dominate the team's time, and a catalog stops being optional (The Data Catalog).
- At 10x team count in the wider organisation: the queue is the company's analytics bottleneck and the ownership axis becomes the only lever left (Data Mesh).
- The visible cost is platform compute and storage, and it is not the interesting one. The dominant cost is the central team's time, and the dominant waste inside that is work that could not start because someone was waiting for an answer about what a field means (Cost Attribution).
- Duplicated computation is low here by construction — one team modelling once is the cheapest arrangement for compute, and it is the reason a central warehouse often looks best on an infrastructure bill while being worst on delivery (Compute Waste).
- Shadow pipelines add cost invisibly: they scan the same sources, hold their own copies and are attributed to no team. They usually appear as an unexplained rise in source-system load rather than in the data platform's own line (What Actually Drives Data Platform Cost).
- The cost of the eventual migration away scales with the number of consumers who hardcoded a table name, which is why publishing through views and a semantic layer is cheap insurance (The Metrics Layer).
- One definition costs delivery speed. Fast delivery costs consistency of meaning. The central warehouse takes the first side of that trade deliberately, and it is the right side for most organisations most of the time.
- Centralising the platform and centralising the modelling look like one decision and are two. Taking both is simple and hits a ceiling; taking only the first is more work up front and does not.
- Absorbing upstream changes centrally is efficient and it removes the producer's incentive to announce them, which makes the next change worse.
Point-to-point, or through the middle
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Point to point | Central log | |
|---|---|---|
| Adding a consumer | A change to every producer it needs data from. | One new subscription. No producer is touched. |
| Schema changes | Negotiated pairwise, differently each time. | Negotiated once, at the topic — which only works if somebody owns the topic. |
| Failure blast radius | One link. Small, and there are many of them. | The log. It is now a dependency of everything, and it must be operated like one. |
| Worth it when | Two or three systems. A broker for two systems is architecture as decoration. | The product n × m has started to hurt, which happens sooner than people expect. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe advantage (one enforcement point for meaning, access and lineage) and the limit (one team's capacity to hold domain semantics) hold regardless of which warehouse product is underneath, and regardless of whether the storage is a database or a table format over files.
- SCALE-SPECIFICThe failure mode is a function of source count and consumer count, not of data volume. A central team comfortably serves petabytes from a handful of systems and struggles with gigabytes from ninety, which is why "we have too much data for a central warehouse" is almost always a misdiagnosis.
- ORG-SPECIFICWhether the queue is tolerable depends on how the company allocates priorities and whether producing teams accept any accountability for their schemas. Two companies with identical technical platforms will experience this arrangement completely differently based on those two facts alone.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the delivery question a central team runs into first: how model changes are tested, deployed and rolled back, and how a queue of change requests is turned into a paved road rather than a review bottleneck.
- — Distributed Systems owns the guarantees the ingestion hops inherit, which is where "one place, one definition" stops helping — a single warehouse fed by an at-least-once stream is a single place holding duplicated rows.