Data Discovery

• Discovery is a ranking problem, not a storage problem. The candidate set is easy — a text match over names, columns and descriptions produces it in milliseconds. Everything that decides whether the tool works happens in the ordering (Linear Search).
• The reason description search fails is that descriptions are the one signal in the system that is optional, written by the wrong person, and never updated. Ranking on them means ranking on the sparsest and least reliable field available (Dataset Documentation).
• The reason usage search works is that querying is not optional. Every SELECT in the warehouse is a costly vote cast by someone who had a reason, and the query log records all of them without anyone deciding to participate (Follow the Query).
• The strongest single signal is distinct recent readers, not query count. Query count is dominated by scheduled jobs, which reflect what was built rather than what is trusted; distinct humans reflects what people choose when they have alternatives (Cardinality: The Label That Took Down Monitoring).
• The second strongest is downstream position: a dataset feeding several dashboards is more likely to be the intended interface than one feeding nothing, and that fact comes from lineage rather than from anything a human wrote (Data Lineage).
• The two failure directions are worth naming. Ranking purely by text finds the plausible name; ranking purely by usage entrenches whatever is popular, including a popular mistake. A usable ranking uses text for recall and evidence for order, with deprecation as a hard demotion rather than a weight (Deprecation as a Process, Not a Label).

Score each dataset by string similarity between the query and its name, description and tags. Return the top matches. Where descriptions are missing, fall back to name similarity, which in practice means the shortest name containing the search term wins.

Use text over names, column names, descriptions and tags to build the candidate set, then order by downstream dashboard count, distinct human readers in a full reporting cycle, and freshness against SLO — with deprecated and staging datasets structurally demoted below everything current regardless of score.

search: "revenue"

  text-only ranking                       evidence ranking
  ────────────────────────────────        ────────────────────────────────
  1. revenue          (deprecated)        1. revenue_daily      12 readers, 4 dashboards
  2. revenue_v2       (0 readers)         2. fct_orders          9 readers, 3 dashboards
  3. revenue_daily                        3. revenue_v2          1 reader, 0 dashboards
  4. tmp_revenue_fix  (1 reader)          4. revenue          [deprecated, demoted]
  5. fct_orders       (no name match)     5. tmp_revenue_fix     1 reader, 0 dashboards
                       ^ ranks last              ^ ranks 2nd via its 'revenue' column

• Index columns, not just tables. The business word a searcher types is far more likely to be a column name than a table name, and this single change usually improves recall more than any amount of description writing (Metadata: Technical, Operational and Business).
• Rank by distinct recent readers and downstream consumer count before text relevance. Text decides the candidate set; evidence decides the order (Data Lineage).
• Demote deprecated and staging datasets structurally rather than by score. A deprecated model should be visibly marked and pushed below everything current, no matter how good its text match (Model Layering).
• Show the evidence on the result row: how many people queried it recently, which dashboards read it, when it last updated, whether its checks pass. A searcher can evaluate a result they can see the basis for (The Data Quality Dashboard).
• Capture zero-result and abandoned searches. They are a free, continuously updated list of what the platform is missing or has named unfindably, and it costs nothing to collect (The Data Catalog).
• Support "who else uses this" as a first-class action. The fastest way for an analyst to validate a table is to see that three people they trust query it weekly, and that is a lineage-and-usage query rather than a documentation one.

• Search guarantees recall only over indexed fields. A dataset with no description and unindexed columns is findable by exact name and by nothing else, which for a new joiner means not findable (The Data Catalog).
• Usage ranking guarantees a reflection of what people queried in the window observed. It does not guarantee they were right — a widely used table with a known flaw ranks first, and popularity is evidence rather than proof.
• Nothing guarantees the searcher understands the result. Discovery ends at "this is probably the right table"; whether they use it correctly is a documentation and grain problem (Grain: What Does One Row Represent?).
• Query-log-derived signals are bounded by log retention and by whether reads through a shared service account can be attributed to individuals. Where they cannot, distinct-reader counts collapse and the strongest signal is lost.