Data Ingestion
Getting data out of systems you often do not control. Batch extracts, incremental windows, streaming producers, and the failure recovery that decides whether a missed hour is recoverable or gone.
Moving data out of systems you do not control and into storage you do — and the difference between what arrived and what happened.
Databases, APIs, logs, files, event streams, SaaS systems and object storage — seven extraction models, seven failure behaviours, and seven different meanings of "everything since last time".
Every hour, select what is new, write files, load the warehouse. The simplest thing that works — and the specific ways it stops working.
Asking a source for "everything since last time" — and the specific, silent, permanent ways `WHERE updated_at > :last_run` gets that wrong.
Event happens, producer publishes, broker durably holds, consumer reads, storage lands it. Continuous rather than windowed — and continuously running.
Not old versus new. Two designs with different freshness shapes, different failure surfaces, different recovery stories and very different operational burdens.
What to do when an extract fails, a connector stalls or a consumer falls behind — and how to tell, quickly, whether the data is late or gone.
Land what arrived, exactly as it arrived, including the fields you do not use. Never clean in place. Partition by arrival so a re-run is bounded.