Evaluating What the Search Returned
A search returns claims, not answers. Four checks turn a claim into something you can build on: is the source authoritative for this tool, is it current, does it match your version, and does it explain why. A result that fails the fourth is a patch, and the first three decide whether it is even a correct one.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Your search returned a confident answer with a code block. Before you use it, what do you need to know about where it came from, and how do you find out in a minute?
The file-upload service is rejecting large uploads with an error you have never seen. A search finds a highly voted answer with a config snippet, a blog post with a different snippet, and an issue thread that seems to be about the same thing but is long. The voted answer looks cleanest. You are about to paste it.
Take the highest-voted or first result. Votes are other people's evaluation, and the first result is the search engine's; delegating the judgment to them feels efficient and the snippet looks like it will work.
The voted answer was correct for a previous major version and its snippet is silently ignored by the current one. The upload still fails, and now there is a config line nobody understands and nobody will remove.
- The voted answer was correct for a previous major version and its snippet is silently ignored by the current one. The upload still fails, and now there is a config line nobody understands and nobody will remove.
- The blog post's snippet works and explains nothing, so the same error in the next service — with a slightly different cause — is a fresh search rather than a recognised mechanism.
- The issue thread, which was the authoritative source and contained the maintainer's explanation and the version in which the behaviour changed, was skipped for being long.
- Two snippets from two sources are both applied "to be safe", and the configuration now has two settings whose interaction nobody has read about.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Treat every result as a claim with a provenance, and run four checks before using it. Authoritative: is the source the tool's own documentation, its issue tracker, a maintainer, or someone whose reasoning you can verify? Current: when was it written, and has the tool changed since? Version: does it name a version, and is it yours? Explains why: does it say what the mechanism is, so that you could tell whether it applies?
- Weight the checks by what you will do with the result. For a one-line fix you will test immediately, version and mechanism matter most; for a design decision you will build on for months, authority and mechanism matter most, and the explanation must be checked against the documentation.
- When results disagree, do not pick; find the mechanism. Two snippets that differ usually differ because the tool changed between versions or because the two authors hit different causes with the same symptom. The issue tracker and the changelog are where that is settled.
- Apply the same four checks to an AI assistant's answer, which is a synthesis of sources it does not cite by default: ask it for the mechanism, then verify the mechanism against the documentation, and treat the version as unknown until confirmed (AI as Debugging Partner).
The result, evaluated three ways
Evaluation is also a question, and it has a quality ladder. The vague form asks whether the result looks right; the better form asks whether it is trustworthy; the best form asks the four checks, and it is the only one that can be answered by looking rather than by feeling.
why The best form is four observations — a byline, a date, a version string, a paragraph — each of which is present or absent on the page. The vague form is answered by how clean the code block looks, which correlates with nothing; the middle form is answered by reputation, which does not survive a major version. Only the best form can rank a long ugly issue thread above a short clean wrong answer, which is the ranking that was needed.
The three results, checked
The matrix is the upload error's three results against the four checks, plus what the check took to run. Notice that the appeal of each result — clean, short, voted — is not a column, and that the result that won looked the least like an answer.
| Result | Authoritative? | Current? | Your version? | Explains why? | Verdict |
|---|---|---|---|---|---|
| Highly voted answer | Community; no maintainer | Years old | Not named; snippet is from the previous major | No — a snippet | Wrong version, silently ignored by yours; discard |
| Recent blog post | An individual; no reasoning shown | Recent | One major behind | "This fixes it" — no mechanism | Current but wrong version; discard |
| Issue thread on the tracker | Maintainer comments near the end | Updated with the current major | Names the version the behaviour changed in | Yes — the limit moved from global to per-route, with a docs link | Use; then confirm on the linked docs page |
| AI assistant's answer | A synthesis; provenance unknown until asked | Unknown | May blend versions | Will explain if asked; must be checked | Ask for the mechanism, verify against the docs, then treat as a lead |
How much checking a result deserves
Not every result deserves all four checks at full weight. The decision is about what you will do with the result: a fix you will test in a minute can survive a weak provenance because the test is the check; a design you will build on cannot. The cost column is what you give up by checking less.
What will I do with this result, and how would I find out it was wrong?
when A one-line change you will run immediately, where a wrong result fails visibly and is reverted in a minute.
cost A result that "works" for the wrong reason may pass the test and fail later; the mechanism check is what catches that.
when A configuration or a pattern you will keep, where a wrong result is silent — ignored settings, a subtly wrong retry policy.
cost A minute per result, and reading the long thread; the payoff is uneven and the habit is what makes it cheap.
when A design decision — a guarantee you will depend on, a library you will build on — where being wrong costs weeks.
cost An afternoon; and it may conclude that the confident results were all wrong and the docs are silent, which is a finding, not a failure (Guarantees and Failure Modes).
when A throwaway experiment in a scratch file where the result is only a hypothesis to test.
cost Nothing, provided it stays in the scratch file; the failure is when the untested result is promoted to the store without the checks ever being run.
How to do it
Most important first.
- Before reading the answer, read the source, the date and any version mentioned. That is a few seconds and disqualifies most wrong results before their code block is tempting.
- Prefer, in order: the docs for your version, the issue tracker, a maintainer, an explained community answer, an unexplained one. Move down the list only when the level above is silent (Search as a Skill).
- Read the long issue thread to the end. The resolution, the version and the mechanism are usually in the last few comments, and the first comment is the same confusion you have.
- For any result you will use, write the mechanism in one sentence of your own before applying it. If you cannot, you do not yet have a result; you have a patch (Explain It Back).
- When two results disagree, search the changelog for the setting or the behaviour. A version boundary usually explains the disagreement.
- After applying, confirm against the documentation for your version that the setting or call means what the result said. That closes the loop from claim to contract (Reading Documentation With a Goal).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- The upload error. Voted answer: dated some years ago, no version named, a config snippet, no explanation. Blog post: recent, names a version one major behind yours, a different snippet, one paragraph saying "this fixes it". Issue thread: on the tool's tracker, long, and in its final comments a maintainer explains that the size limit moved from a global setting to a per-route option in the current major version, and links the docs page. The four checks rank them in reverse order of how appealing they looked: the thread is authoritative, current, your version and explains why; the blog post is current but wrong version; the voted answer is wrong version and explains nothing. The fix is the per-route option from the linked docs page, and the mechanism — the limit is per route now — is a sentence you can say.
- Two disagreeing answers about the chat app's realtime library: one says reconnect manually, one says the client reconnects automatically. The changelog shows automatic reconnection was added in a minor version; the "manual" answer predates it. Neither answer was wrong when written; the version check was the whole evaluation (Resynchronisation After a Gap).
- An AI assistant's confident answer about a payment provider's webhook retry policy. Asked for the mechanism, it gave a plausible one; checked against the provider's docs, the retry schedule it described was for a different provider. The four checks applied to a synthesis found the blend that a search result's byline would have made obvious.
How you know it worked
What now exists that did not before, and what question you can now ask.
- You looked at the source, date and version before you looked at the code block.
- You can say the mechanism in one sentence, and it agrees with the documentation for your version.
- A disagreement between results was resolved by finding a version boundary or a distinct cause, not by picking the cleaner snippet.
- A long authoritative thread was read to the end instead of skipped for a short confident one.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Who wrote this, and are they in a position to know how this tool actually behaves?
- ?When was it written, and what has changed in the tool since?
- ?Does it name a version, and is that version mine?
- ?What mechanism does it claim, and does the documentation for my version agree?
- ?If two results disagree, what version boundary or distinct cause explains the disagreement?
What can go wrong
- Four checks on every result, including for a trivial question with a low-stakes answer. Weight by what you will do with it; a one-liner you will test in a minute needs the version check and little else.
- Authority as a substitute for mechanism: the docs said it, so it is applied without understanding, and the next version changes it. The docs are the best source of the mechanism, not a replacement for holding it.
- Over-trusting recency: the newest blog post is assumed to be for the newest version, when it may be a recent post about an old setup. Date and version are separate checks.
- Rejecting community answers entirely because they are not authoritative. An explained community answer that you verify against the docs is a good result; the failure is using it without the verification.
- The checks take a minute per result, and on a question with a dozen plausible results that is a dozen minutes before any fix is tried.
- Preferring authoritative sources means reading terser, less friendly material than the community answer that would have gotten you unstuck faster — and might have been right.
- Reading the long thread to the end is slow and often the thread turns out to be about something else; the cost is real and the payoff is uneven.
- "Votes are a good enough proxy." Votes measure how many people the answer helped at the time. They are a decent proxy for correctness on the version that was current then, and no proxy at all for yours.
- "The docs are the only trustworthy source." The docs are the trustworthy source for intent. The issue tracker is often more trustworthy for behaviour, because that is where the gaps between intent and behaviour are recorded.
- "An AI answer that cites sources has passed the checks." A citation is a claim of provenance; it has to be opened, and the four checks run on what is found there. A citation that does not say what the answer said it does is the most common failure of the form.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAuthoritative, current, your version, explains why apply to any claim from any medium — a search result, a colleague's memory, a conference talk, an AI assistant — because they are questions about provenance, not about search engines.
- ILLUSTRATIVEThe upload error, the three results and the maintainer's comment are invented to show the shape of the evaluation; no real tool's version history is being described.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's "Review the LLM's answer" at /manifesto/review is the same four checks turned on a synthesis: provenance, currency, version, mechanism — and the docs as the thing the answer is checked against, not replaced by.