Problem says X → think Y
The searchable index of this domain. The left column is what the problem sounds like when somebody says it out loud in a meeting; the right column is the mechanism to think of before you open a query editor.
63 of 63 rows
| The problem says | Think |
|---|---|
| Analytics queries are slowing down production | OLTP and OLAP are different workloads on the same storage. Separate them before it is urgent.Workload Isolation → |
| We need to analyse five years of history | The operational schema overwrites history. You need an analytical store and a model that preserves it.OLAP Workloads → |
| We need somewhere cheap to keep everything | Object storage as a raw layer, immutable, partitioned by arrival.The Raw Landing Zone → |
| Queries read every column and use two | Columnar storage plus projection pushdown. This is the cheapest optimisation available.Projection Pushdown → |
| The query scans far more than it returns | Partition pruning — and check the predicate is one the planner can actually see through.Partition Pruning → |
| A one-day query reads the whole year | The filter does not match the partition key, or it is wrapped in a function.Partition Pruning → |
| Millions of tiny files | Compaction. On object storage the cost is per request, so the pathology is file count, not size.File Compaction → |
| Listing a table takes longer than reading it | Partition cardinality is too high. Fewer, larger partitions.Partition Cardinality → |
| We partitioned by user_id and everything got slower | A high-cardinality partition key moves the cost into metadata.Partition Cardinality → |
| We need database changes as they happen | Change data capture, reading the transaction log rather than polling the table.Change Data Capture → |
| The nightly extract never captures deletes | A timestamp predicate structurally cannot see a row that no longer exists. CDC can.CDC vs Polling → |
| Rows are missing and nobody knows when they went | Reconcile against the source for a closed period, then walk upstream hop by hop.Missing Rows → |
| Many teams want the same events | A durable partitioned log, read independently, rather than a connector per consumer.The Event Log → |
| We need to reprocess from the beginning | Replay from a retained log, or recompute from an immutable raw layer. Retention is the boundary on both.Replay from the Log → |
| Events arrive after the day is closed | Event time, watermarks and an explicit allowed-lateness decision.Late Events → |
| Yesterday's number changed overnight | Late-arriving data revising a closed period. Decide whether periods are provisional or final.Late-Arriving Data → |
| The same order appears twice | At-least-once delivery is the normal case. Deduplicate on a business key, and expect an id-based dedup to miss producer retries.Deduplication → |
| Re-running the pipeline duplicated the output | The pipeline is not idempotent. Replace the partition or merge on the business key.Idempotent Data Pipelines → |
| One Spark task takes forever and the rest finish immediately | Data skew. Adding workers will change nothing.Data Skew → |
| The job is slow and the CPU is idle | A shuffle. Network, spill and serialization, not computation.The Shuffle → |
| Adding executors did not help | The job is bound by its slowest task, or by a partition count that caps parallelism.Straggler Tasks → |
| A join makes the row count explode | Grain. One side is not unique on the join key.Grain: What Does One Row Represent? → |
| Revenue is exactly double | A fan-out join, or a re-run that appended instead of replacing.Duplicate Rows → |
| What does one row of this table mean? | That is the grain, and if it is not written down the table cannot be safely aggregated.Grain: What Does One Row Represent? → |
| We need the customer's country as it was at order time | A Type 2 slowly changing dimension with validity ranges.SCD Type 2 in Practice → |
| Historical reports changed when a customer moved | A Type 1 dimension is overwriting history the report depends on.Slowly Changing Dimensions → |
| We need the balance on each past day | A snapshot table. State is not reconstructable from events unless every event is retained.Snapshot Tables → |
| The dashboard is stale | Freshness per dataset, monitored against a stated SLO — not per platform.Freshness Monitoring → |
| All jobs are green and the data is wrong | Task status cannot see missing rows, duplicates, late data or semantic drift. Monitor the data.The Pipeline Succeeded. The Data Is Wrong. → |
| A metric dropped 40% overnight | Walk upstream: serving table, transformation, upstream dataset, ingestion, source.Debugging a Data Incident → |
| Two dashboards disagree about revenue | The metric is defined in two places. It needs one definition with an owner.The Metrics Layer → |
| Every team computes it slightly differently | A metrics layer, not another dashboard.The Metrics Layer → |
| A column rename broke eleven dashboards | Data contracts at the boundary, plus lineage so impact is knowable before the change.Data Contracts → |
| The schema check passed and the numbers are wrong | A semantic change. The type is unchanged and the meaning is not.Semantic Changes → |
| A field became a string and revenue went to zero | The cast produced null rather than an error. Enforce the contract at the boundary.Breaking Schema Changes → |
| Adding a nullable field broke a consumer | Compatibility is a property of the serialization and the consumer's assumptions, not of the change alone.Backward Compatibility → |
| If I change this column, what breaks? | Impact analysis over the lineage graph, downstream direction.Impact Analysis → |
| Where did this number come from? | Column-level lineage, walked upstream from the metric.Where Did This Number Come From? → |
| Nobody knows who owns this table | Ownership is the precondition for every other governance control.Data Ownership → |
| A bug has been wrong for six months | Fix forward, then backfill the affected range — with a bounded range and validation before publishing.Planning a Backfill → |
| The backfill corrupted the current partition | It appended rather than replacing, or it ran against a source that has since changed.What Backfills Break → |
| Consumers saw half a dataset | Atomic publish: write elsewhere, validate, then swap.Atomic Publish → |
| Two of a hundred partitions failed | Re-run only the failed unit — which is safe only if the unit is idempotent and independently publishable.Partial Failure → |
| The full rebuild no longer finishes overnight | Incremental processing, with an explicit answer for late data.Incremental Processing → |
| The watermark skipped rows that committed late | A wall-clock timestamp is not monotonic under concurrency. Use a log position or an overlapping window.The High-Water Mark → |
| The warehouse bill doubled | Bytes scanned, bytes shuffled, bytes retained, or history being recomputed for no reason.What Actually Drives Data Platform Cost → |
| One query costs more than the rest of the platform | It scans without pruning. Column selection and a matching partition key.Scan Cost → |
| We rebuild everything nightly | Cost scaling with history rather than with new data.Compute Waste → |
| Storage keeps growing and nothing is ever deleted | A lifecycle policy, argued as a recovery-window and liability decision rather than a cost one.Storage Lifecycle → |
| Should this be streaming? | Only if a decision genuinely cannot wait. Compare freshness, complexity, cost, failure handling and operational burden.Batch vs Streaming Ingestion → |
| We want exactly-once | Separate input consumption, state update and output write. Then name which one you mean.Exactly-Once: Input Consumption, State Update, Output Write → |
| A consumer fell behind and the log expired | Retention is a recovery deadline, not a storage setting.Retention and Replay → |
| We added consumers and throughput did not change | Partition count is the hard ceiling on consumer-group parallelism.Consumer Groups and the Parallelism Ceiling → |
| PII appeared in an analytics table | The copy inherited the obligation and none of the enforcement. Classify at ingest, minimise, and mask.PII in Pipelines → |
| A deletion request arrived | Deleting the source row deletes nothing downstream. Design for it before the audit.Deletion Requests → |
| Analysts should only see their region | Row-level policy enforced by the store, not by convention in every query.Row and Column Security → |
| Nobody can find the right dataset | A catalog with owners, freshness and usage — and populated, or it is worse than nothing.The Data Catalog → |
| Every team built its own ingestion | A platform problem, not a pipeline problem.Data Platform Engineering → |
| The central data team is the bottleneck | Domain ownership with a self-service platform — and the federated governance that has to come with it.Data Mesh → |
| We changed the embedding model | That is a data migration: re-embed the corpus, build a new index, and switch.Re-embedding → |
| Retrieval quality dropped after a re-index | Chunking, embedding version and source version are pipeline state that must be tracked together.Vector Data Engineering → |
| We need an evaluation set from production traffic | A sampling and privacy-filtering pipeline, with the same lineage and freshness discipline as any other dataset.Evaluation Data Pipelines → |
| Which warehouse should we use? | Name the capability you lack first. Most warehouse projects are launched to solve something a layout change would fix.Choosing an Analytical Platform → |