Data Engineering and Cloud Infrastructure
A data platform is assembled almost entirely from four cloud primitives. Knowing which four, and what each actually charges you for, is most of platform engineering.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which cloud primitives is a data platform actually built from, and which decisions about them are irreversible?
The team that will operate this platform for the next five years, and the finance partner who will ask in month nine why the storage line grows every month and never falls. Both need to know which cost is a consequence of a design decision and which is a consequence of a volume they do not control (Cost Attribution).
The unit that crosses this boundary is one object in a bucket — an immutable, versioned, individually-addressable blob with a key that looks like a path but is not one. Almost everything distinctive about lakes, table formats, partitioning and small-file problems follows from the properties of that single unit (Object Storage as Data Infrastructure).
Pick a managed warehouse, load everything into it, and let the provider handle storage, compute, scaling and backups. This genuinely is the right first answer for most organisations, and treating it as naive would be a mistake — it is the default because it works (The Data Warehouse).
A streaming ingest writes one small object per micro-batch per partition. The bytes are trivial and the file count is not: every query now lists and opens an enormous number of objects, and listing is the bottleneck rather than reading (File Size and the Small-Files Problem, File Compaction).
- A streaming ingest writes one small object per micro-batch per partition. The bytes are trivial and the file count is not: every query now lists and opens an enormous number of objects, and listing is the bottleneck rather than reading (File Size and the Small-Files Problem, File Compaction).
- A job in one region reads a bucket in another. It works, it is not obviously slow, and it accrues cross-region transfer on every run forever (Egress: Moving Data Costs Money, Not Just Storing It, Multi-Region Deployment).
- Everything in the pipeline runs as one identity with broad access, because that was what got the first job working. Two years later there is no way to answer "which pipeline can read the PII bucket" (The IAM Model, Least Privilege in Infrastructure).
- A lifecycle rule moves objects to a colder tier after ninety days. Nine months later a backfill needs a year of history and every read pays a retrieval penalty and a latency the job was never designed for (Storage Lifecycle, Planning a Backfill).
- The warehouse's compute is left running between scheduled jobs because someone disabled auto-suspend during an incident, and the platform is paying for held capacity that is doing nothing (Compute Waste, Idle Capacity: Headroom or Waste?).
- A serverless transformation opens a connection to the source database per invocation, and a burst of concurrency exhausts the database's connection pool (Serverless and Database Connections, Connection Pools).
What is actually happening
- Object storage is not a filesystem, and every difference matters. Objects are immutable — a change writes a new object. There are no directories, only key prefixes and a listing API. There is no atomic multi-object operation, which is exactly the gap that open table formats exist to fill (Direct Uploads and Signed Authorization, Open Table Formats).
- Separating storage from compute is an availability and elasticity property before it is a cost one. Data outlives any cluster, several engines can read the same bytes, and compute can be sized per workload. The tax is that every read crosses a network that a local disk did not (Separating Storage from Compute).
- Compute comes in three shapes and the choice is about the workload's duty cycle: always-on instances for continuously running stream jobs, scheduled or autoscaled clusters for batch, and per-invocation serverless for small, spiky, stateless work. Matching the shape to the duty cycle is the whole decision (Choosing a Compute Model, Serverless Trade-offs).
- IAM is the data platform's access control layer whether or not anyone designed it that way. A bucket policy and a role are what actually decide who can read the raw PII landing zone; warehouse grants only govern what is inside the warehouse (Anatomy of a Policy, Data Access Control).
- Networking decides where data can go and what movement costs. Private connectivity, VPC boundaries and egress charges are why "just read it from the other account" turns into a project, and they are also the enforcement point for keeping a dataset inside a jurisdiction (Cloud Networking: Composing the Fundamentals, Private Connectivity).
- Managed services move operational burden, not architectural responsibility. The provider runs the storage engine; grain, layout, freshness, quality and access remain entirely yours, and none of them appear on a status page (Shared Responsibility).
- Key management is where encryption becomes a data-engineering concern. Encryption at rest is nearly free and nearly automatic; deciding which key encrypts which dataset is what makes crypto-shredding a viable deletion strategy later (Key Management and Encryption at Rest, Deletion Requests).
Where the boundary sits
Cloud & Infrastructure asks where things run, how they are provisioned, and how the underlying resources behave. This domain asks which of those primitives a data platform is made of and what each one implies about layout, freshness, access and cost. We use four of them heavily — object storage, compute, identity and networking — and the depth on all four lives next door.
The most useful habit on this boundary is to translate every provider concept back into the primitive before reasoning about it. "A warehouse" is a query engine with attached or detached storage. "A managed streaming service" is a partitioned log with a retention policy. Once translated, the design questions are the ones this domain already answers, and the product name only decides the configuration.
The third column below is where the primitive stops being infrastructure and starts being a data decision.
| We teach | Depth lives in | The mechanism that crosses |
|---|---|---|
| Laying data out as files in a bucket | Cloud: Object Storage, Direct Uploads and Signed Authorization | Objects are immutable, keys are not paths, and there is no atomic multi-object write. That single set of properties produces the small-file problem and the need for table formats. |
| Choosing where transformation runs | Cloud: Choosing a Compute Model, Serverless as an Execution Model | Duty cycle decides the shape. Always-on for streams, elastic for batch, per-invocation for spiky small work (Where the Transformation Actually Runs). |
| Deciding who can read a dataset | Cloud: The IAM Model, Anatomy of a Policy | The bucket policy is the real access control for anything outside the warehouse. Warehouse grants stop at the warehouse edge (Data Access Control). |
| Keeping cost proportional to value | Cloud: Cost Engineering, Right-Sizing Without Causing an Outage | Retained bytes, held compute and transferred bytes are the three drivers, and each is set by a design decision rather than by usage (What Actually Drives Data Platform Cost). |
| Deciding what history stays warm | Cloud: Storage Lifecycle, Backup Strategy | A lifecycle rule is a bet about your future backfill range. Getting it wrong is discovered during an incident (Planning a Backfill). |
| Placing data in a region | Cloud: Regions and Availability Zones, Multi-Region Deployment, Cloud Networking: Composing the Fundamentals | Region is a failure boundary, a legal boundary and a transfer-cost boundary simultaneously, and the three rarely point the same way. |
| Making the platform reproducible | Cloud: Infrastructure as Code, State: The File That Makes It Work and the File That Will Hurt You | A data platform is buckets, roles, warehouses and schedules. If those are not code, no environment can be recreated and no rollback is real (Rolling Back Data). |
| Encrypting datasets | Cloud: Key Management and Encryption at Rest, Secrets in Infrastructure | Encryption at rest is table stakes; key *scope* is the data decision, because it determines whether deletion can be done by destroying a key (Data Masking, Tokenisation & Encryption). |
What actually drives a data platform bill
Cost on this boundary is not a finance topic, it is a design topic, and the reason it belongs in a connections lesson is that every driver below is set by a decision made in another domain's vocabulary. A partition key is a layout decision that shows up as scanned bytes. A retention policy is a recovery decision that shows up as retained bytes. An identity design shows up as nothing at all until an audit.
The weights below are relative and unitless — they establish an ordering for a typical warehouse-plus-lake platform, and they are not measurements and not transferable to any specific environment. The ordering is the teaching: the drivers people optimise first are usually the smallest.
The pattern worth internalising is that the two largest drivers are both consequences of layout and scheduling, which are decided early, cheaply, and by whoever wrote the first pipeline.
Set by partitioning, file layout, column projection and how much history each model re-reads. It is the driver a layout change moves most (Scan Cost).
A warehouse that does not suspend, a cluster kept warm for convenience, a stream job over an idle topic. Produces cost with no work and raises no alarm (Idle Capacity: Headroom or Waste?).
Grows monotonically because nothing ever deletes by default. Includes snapshots, object versions and abandoned experiments (Storage Lifecycle).
A nightly full rebuild costs in proportion to all history rather than to the day that changed (Incremental Processing).
Usually small and usually accidental — the driver most likely to be a configuration mistake rather than a design choice (Egress: Moving Data Costs Money, Not Just Storing It).
Negligible with sensibly sized files and dominant once a bucket holds a very large number of tiny objects, which is why file size is a cost control (File Size and the Small-Files Problem).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative, unitless, and shown to establish an ordering rather than a magnitude. The teaching is that layout and scheduling dominate, and the line that looks alarming on an invoice is rarely the one worth fixing first.
Choosing where the transformation runs
This is the decision that new data platforms get wrong most often, usually by picking the compute shape the team already knows rather than the one the workload's duty cycle asks for. The criteria are the lesson; there is no winner.
The question to ask first is not "how much data" but "what fraction of the time is this thing doing work". A job that runs for two minutes an hour and a job that never stops have opposite optimal shapes, at any volume.
The second question is what the job holds. Anything with keyed state that must survive restarts pushes you towards long-lived compute with a checkpoint story, regardless of how little data it processes (Streaming State).
What is this job's duty cycle, and does it hold state between records?
when The data is already in the warehouse, the transformation is expressible as SQL, and you want the optimiser, the statistics and the governance to be someone else's job. This is the default and it deserves to be (ELT: Load First, Transform Where the Data Lives).
cost Compute is billed by the product's own dimension and is easy to waste; you are bound to its dialect and its engine; non-SQL work has nowhere to go (SQL Transformations).
when Large scans, wide shuffles, non-SQL logic, or data that lives in files rather than tables. Scheduled runs with a clear start and end (Distributed Data Processing).
cost Cluster startup latency per run, real tuning work — partitions, skew, memory — and an idle cost if the cluster is kept warm to avoid the startup (Data Skew).
when Continuous processing with keyed state, event-time windows and a freshness requirement that a schedule cannot meet (Stream Processing).
cost Paid continuously whether or not events arrive; state must be checkpointed and bounded; it is a permanently running production system with its own on-call (Checkpointing).
when Small, spiky, stateless work — a webhook landing, a file-arrival trigger, a light per-event enrichment (Serverless as an Execution Model).
cost Cold-start latency, execution-duration limits, and concurrency that can overwhelm a downstream database's connection pool at exactly the moment traffic spikes (Serverless and Database Connections).
when The dataset genuinely fits, the work is exploratory or a small scheduled model, and distributed machinery would be pure overhead (DuckDB Concepts).
cost A ceiling you will hit without warning, and a habit of local scripts that never become reproducible pipelines (Data Engineering Anti-Patterns).
Product names, tiers, auto-suspend defaults, concurrency limits and execution-duration caps all change. Treat the shapes above as stable and verify every specific limit against current provider documentation before designing against it.
How to build it
Most important first.
- Design the bucket and prefix layout as deliberately as a schema. It is the boundary for access policy, for lifecycle rules, for cost attribution and for partition pruning, and it is very hard to change once pipelines reference it (Physical Data Layout, Cost Attribution).
- Give every pipeline its own workload identity with the narrowest access that lets it run, and never a long-lived static key. "Which job can read this" should be answerable from a policy, not from tribal memory (Roles vs Static Keys, Human vs Workload Identity).
- Keep storage and compute in the same region as the data, and treat any cross-region read as a design decision with a written reason (Egress: Moving Data Costs Money, Not Just Storing It).
- Choose the compute shape from the duty cycle, not from familiarity: a job that runs for two minutes an hour and a job that never stops want opposite things (Choosing a Compute Model).
- Write lifecycle policies at the same time as the pipeline that fills the bucket, and pick tiers against your realistic backfill range rather than against average access (Storage Lifecycle, Data Retention).
- Express the whole platform as infrastructure code, so a second environment is a parameter change and drift is visible rather than discovered (Infrastructure as Code, Drift: When the File and Reality Disagree).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Object storage guarantees very high durability of a stored object and read-after-write visibility for a new object. It guarantees nothing about the consistency of a *set* of objects, which is why a multi-file publish is not atomic without a table format (Atomic Publish).
- A managed warehouse guarantees transactional semantics inside itself. It guarantees nothing about the completeness of what was loaded into it, which is the recurring lesson of this whole domain.
- A region guarantees a failure-isolation boundary. Multi-zone protects against a facility, multi-region against a region, and neither protects against a pipeline writing wrong data everywhere at once (Failure Domains, Multi-Zone Deployment).
- A managed service guarantees the provider's side of the shared-responsibility line and nothing on yours. Backups of the platform are not backups of your data model (Shared Responsibility, Backup Strategy).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that belongs here is an inventory reconciliation: object count and total bytes per partition prefix against what the pipeline's manifest or table metadata claims should exist.
- It catches partial publishes, orphaned files from failed jobs, and a lifecycle rule that removed objects a table still references. It misses anything about the *contents* of those objects — every file can be present and every row inside them wrong.
- The complementary check is an access-policy assertion: enumerate which identities can read each classified prefix and diff it against the intended list. It catches privilege creep and misses whoever already copied the data out (Data Access Control).
- Object storage adds a network hop to every read that a local disk did not, which shows up as higher per-request latency and is irrelevant to a batch scan and highly relevant to a point lookup (Object Storage as Data Infrastructure).
- Serverless compute adds cold-start latency per invocation, which is invisible in a nightly batch and dominant in a per-event handler (Startup Time & Cold Start).
- Cross-region reads add round-trip latency that is a physical property of distance, and no amount of provisioning removes it (Cross-Region Latency Is Physics, Not Configuration).
- Bucket and prefix layout is the hardest thing on this boundary to change, because every pipeline, policy, lifecycle rule and external consumer references it by literal path. Treat it as a public interface from day one.
- Storage class and lifecycle rules can be changed at any time and apply going forward; what cannot be undone cheaply is a transition that has already moved a large history into a colder tier.
- Provider primitives are architecturally stable over years — an object store, a managed warehouse, a queue — while their service names, tiers and feature sets are not. Design against the primitive and keep the product name in one configuration layer (Cloud Data Services).
- Object versioning and soft-delete windows are the cheapest insurance in a data platform, and they only help if they were enabled before the incident (Backup Strategy).
- Recovery of a table format's state is a metadata operation — roll back to a previous snapshot — which is far cheaper than restoring objects, and is one of the strongest arguments for a table format over bare files (Open Table Formats, Rolling Back Data).
- A restore that has never been tested is a hypothesis. Testing it against a real dataset, at real size, is the only way to learn that the retrieval tier makes it take an order of magnitude longer than anyone assumed (Restore Testing, RPO & RTO).
What can go wrong
- Small-file proliferation turning a cheap scan into a listing and open-request problem.
- A cross-region read added for convenience and paid for on every run thereafter.
- A single over-broad identity used by every pipeline, making access impossible to audit and impossible to narrow later.
- A lifecycle transition that makes a needed backfill slow, expensive and surprising.
- The mitigation failing: object versioning enabled but never lifecycle-expired, so the "safety net" quietly becomes the largest object in the storage bill.
- Compute left held between jobs, which produces cost with no corresponding work and no error anywhere (Compute Waste).
- "Storage is cheap, so keep everything." Retained bytes are the one driver that only ever grows, and the cost that actually bites is not the storage — it is the scan, the copy, the governance obligation and the deletion request that now has to search six layers (Data Minimization).
- "Separating storage and compute makes it cheaper." It makes it elastic and portable. Whether it is cheaper depends entirely on the duty cycle: continuously busy compute is not obviously cheaper for being detached.
- "The provider handles backups." The provider protects its infrastructure from its failures. It does not protect your table from your pipeline, and a bad transformation replicated across three zones is replicated correctly (Shared Responsibility).
- "Cloud services map one-to-one across providers." The primitives rhyme and the semantics differ — consistency, ordering, quotas, identity models and lifecycle behaviour all vary, and a design that assumed equivalence breaks at exactly the point that matters (Cloud Data Services).
- "Serverless removes capacity planning." It moves it. Concurrency limits, cold starts and downstream connection pools become the capacity constraints instead (Serverless and Database Connections).
- The bucket, the prefix and the IAM role are where data classification becomes enforceable. A classification that is documented but not reflected in a policy is a label, not a control (Data Classification).
- Key scope decides whether crypto-shredding is available as a deletion mechanism later. Deciding to encrypt per tenant or per subject is cheap on day one and effectively impossible to retrofit (Key Management and Encryption at Rest, Deletion Requests).
- Region choice is frequently a legal constraint rather than a latency one, and it applies to every copy — including a backup, a replica and an analyst's extract (Data Governance).
Operating it
- Object count and average object size per partition prefix, trended. It is the earliest signal of a small-file problem and it is almost never monitored (File Size and the Small-Files Problem).
- Bytes scanned or compute seconds per pipeline and per consumer, tagged so the cost can be attributed to a dataset rather than to the platform (Cost Attribution, Cost per Service and the Attribution Problem).
- Cross-region and cross-account transfer volume, broken out, because it is the driver most likely to be a mistake rather than a choice (Egress: Moving Data Costs Money, Not Just Storing It).
- Access logs on classified prefixes, retained and reviewed — the only record of who actually read the data rather than who could (Audit Trails, Audit Logs for Privileged Actions).
- At 10x data volume the layout decisions start dominating and the platform's cost stops being proportional to volume and starts being proportional to how carelessly it is queried.
- At 100x, or at many teams, the binding constraint is usually concurrency and isolation rather than storage: who gets compute during the month-end close, and whose exploratory query can slow the pipeline (Workload Isolation).
- Account and project structure scales the governance problem. One account is simple and unauditable; per-domain accounts are auditable and require a real identity design (Data Mesh, The IAM Model).
- Retained bytes across every layer, including the copies nobody deleted: raw, staging, curated, snapshots, versioned objects and the extract in a notebook (Storage Lifecycle).
- Bytes scanned by queries, which is the driver a layout decision moves most and the one people try to fix last (Scan Cost).
- Compute held rather than compute used — a cluster kept warm, a warehouse that did not auto-suspend, a stream job running over an idle topic (Compute Waste, Idle Capacity: Headroom or Waste?).
- Request and transfer volume: listing operations against a bucket full of small objects, and any byte that crosses a region or an account boundary (Egress: Moving Data Costs Money, Not Just Storing It).
- Managed services buy operational leverage and cost portability. Your data is reachable only through the provider's engine, and the migration path is a project rather than a copy (Lake vs Warehouse vs Lakehouse).
- Open formats on object storage buy portability and cost you the maintenance work — compaction, expiry, statistics — that a managed warehouse does invisibly (The Lakehouse).
- Fine-grained identities buy auditability and cost real ongoing effort; every new pipeline is now a policy change, and teams route around policy friction unless the path is paved (The Self-Service Data Platform).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALObject immutability, the absence of directories, the lack of multi-object atomicity, and the separation of storage from compute are architectural properties of every major provider and have been stable for years.
- CLOUD-SPECIFICConsistency, listing behaviour, identity models and transfer boundaries differ enough that a design ports in shape but not in detail: an IAM policy attached to a role is not the same object as a resource-scoped grant, and cross-account access is a different mechanism from cross-project access. Treat any 1:1 mapping table as a starting point, never as an equivalence (Cloud Data Services).
- SCALE-SPECIFICBelow a few pipelines and one team, a single account, one bucket and a shared role is the correct amount of structure and per-domain accounts are pure overhead. The advice inverts once more than one team can write to the same prefix or once a dataset carries a classification.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the delivery half of infrastructure-as-code — pipelines that plan and apply, environment promotion, and what a rollback of infrastructure means when the thing it provisioned holds data.