Skip to content

Engineering · For your bots

Docs versus code

Find statements in the docs that the code now contradicts: quote the doc, show the code, and say which of the two is wrong.

In Lobstack: Skills › Library › Add

Download Lobstack

When to use it

A README, guide or API reference may have fallen behind the code, or a release changed behaviour and nobody checked the pages that describe it.

What your bot will do

  1. 01

    Pick out the statements that can be checked: install steps, commands, flags, config keys, defaults, API fields, limits, error messages. Skip background and opinion.

  2. 02

    For each one, find and read the code that decides it — the default value, the option parser, the handler. Another doc page is not a check.

  3. 03

    Run what is safe to run: install steps in a scratch folder, the help output, a read-only API call. A documented command that fails is the strongest finding there is.

  4. 04

    For each disagreement, quote the doc line, show the code line with its file, and say which one is wrong. Sometimes the doc is the intent and the code is the bug.

  5. 05

    Ignore wording, tone and formatting. Only false statements count.

  6. 06

    List documented features you cannot find in the code at all. A page for something that no longer exists is the costliest kind of drift.

  7. 07

    Post a document, most misleading first. Offer the fixes on a branch, and ask before pushing or opening a pull request.

Quote both sides every time, so a reader can settle it without trusting you.

The file

docs-drift-report/SKILL.md34 lines
---name: docs-drift-reportdescription: "Find statements in the docs that the code now contradicts: quote the doc, show the code, and say which of the two is wrong."metadata:  title: "Docs versus code"  category: engineering  tags: [docs, accuracy]--- ## When to use this A README, guide or API reference may have fallen behind the code, or a releasechanged behaviour and nobody checked the pages that describe it. ## How 1. Pick out the statements that can be checked: install steps, commands, flags,   config keys, defaults, API fields, limits, error messages. Skip background   and opinion.2. For each one, find and read the code that decides it -- the default value,   the option parser, the handler. Another doc page is not a check.3. Run what is safe to run: install steps in a scratch folder, the help output,   a read-only API call. A documented command that fails is the strongest   finding there is.4. For each disagreement, quote the doc line, show the code line with its file,   and say which one is wrong. Sometimes the doc is the intent and the code is   the bug.5. Ignore wording, tone and formatting. Only false statements count.6. List documented features you cannot find in the code at all. A page for   something that no longer exists is the costliest kind of drift.7. Post a document, most misleading first. Offer the fixes on a branch, and ask   before pushing or opening a pull request. Quote both sides every time, so a reader can settle it without trusting you.

Get Lobstack.