Data Architecture Patterns

Central warehouse, event-driven platform, Lambda, Kappa and mesh — sorted onto the two independent axes they actually live on, and compared by the problem each was a response to.

The arrangement most organisations actually run, taken seriously: one team, one place, one definition — with a real advantage and a specific failure mode that arrives with source count rather than with data volume.

Everything publishes events; consumers subscribe independently. It buys decoupling, replay and many materialisations of one stream — and it moves duplicate, ordering and schema handling from one place into every consumer.

A batch layer that is authoritative but late, a speed layer that is fresh but provisional, and a serving layer that merges them — bought with two implementations of the same logic that must agree forever.

One event log, one stream processing path, and reprocessing by replay. It removes Lambda's duplicated implementation and replaces it with two demands: the log must retain everything you might reprocess, and the stream job must replay history at a rate batch used to manage.

An organisational model, not an architecture: domain ownership, data as a product, a self-service platform and federated governance — with the operational cost of each stated honestly.

Owner, schema, semantics, quality, documentation, SLO, access policy. Seven commitments, and what a team has to start doing on the day it makes them.

Eight shared capabilities — ingestion, storage, compute, orchestration, catalog, quality, security, observability — and the boundary question that decides whether the platform team is a substrate or a queue.

An engineer declares a source and a model; the platform produces a pipeline, tests and monitoring. Get it wrong in one direction and it is a ticket queue with extra steps; get it right and you have four hundred datasets nobody owns.